docs: Tutorial - Using go-ipfs as a Library

.gitignore

@@ -25,3 +25,4 @@ bin/cover
 vendor
 .tarball
 go-ipfs-source.tar.gz
+docs/examples/go-ipfs-as-a-library/example-folder/Qm*

docs/examples/go-ipfs-as-a-library/README.md

+# Use go-ipfs as a library to spawn a node and add a file
+
+> This tutorial is the sibling of the [js-ipfs IPFS 101 tutorial](https://github.com/ipfs/js-ipfs/tree/master/examples/ipfs-101).
+
+By the end of this tutorial, you will learn how to:
+
+- Spawn an IPFS node that runs in process (no separate daemon process)
+- Create an IPFS repo
+- Add files and directories to IPFS
+- Retrieve those files and directories using ``cat`` and ``get``
+- Connect to other nodes in the network
+- Retrieve a file that only exists on the network
+- The difference between a node in DHT client mode and full DHT mode
+
+All of this using only golang!
+
+In order to complete this tutorial, you will need:
+- golang installed on your machine. See how at https://golang.org/doc/install
+- git installed on your machine (so that go can download the repo and the necessary dependencies). See how at https://git-scm.com/downloads
+- IPFS Desktop (for convenience) installed and running on your machine. See how at https://github.com/ipfs-shipyard/ipfs-desktop#ipfs-desktop
+
+
+**Disclaimer**: The example code is quite large (more than 300 lines of code) and it has been a great way to understand the scope of the [go-ipfs Core API](https://godoc.org/github.com/ipfs/interface-go-ipfs-core), and how it can be improved to further the user experience. You can expect to be able to come back to this example in the future and see how the number of lines of code have decreased and how the example have become simpler, making other go-ipfs programs simpler as well.
+
+## Getting started
+
+**Note:** Make sure you have [![](https://img.shields.io/badge/go-%3E%3D1.13.0-blue.svg?style=flat-square)](https://golang.org/dl/) installed.
+
+Download go-ipfs and jump into the example folder:
+
+```
+> go get -u github.com/ipfs/go-ipfs
+cd $GOPATH/src/github.com/ipfs/go-ipfs/docs/examples/go-ipfs-as-a-library
+```
+
+## Running the example as-is
+
+To run the example, simply do:
+
+```
+> go run main.go
+```
+
+You should see the following as output:
+
+```
+-- Getting an IPFS node running --
+Spawning node on a temporary repo
+IPFS node is running
+
+-- Adding and getting back files & directories --
+Added file to IPFS with CID /ipfs/QmV9tSDx9UiPeWExXEeH6aoDvmihvx6jD5eLb4jbTaKGps
+Added directory to IPFS with CID /ipfs/QmdQdu1fkaAUokmkfpWrmPHK78F9Eo9K2nnuWuizUjmhyn
+Got file back from IPFS (IPFS path: /ipfs/QmV9tSDx9UiPeWExXEeH6aoDvmihvx6jD5eLb4jbTaKGps) and wrote it to ./example-folder/QmV9tSDx9UiPeWExXEeH6aoDvmihvx6jD5eLb4jbTaKGps
+Got directory back from IPFS (IPFS path: /ipfs/QmdQdu1fkaAUokmkfpWrmPHK78F9Eo9K2nnuWuizUjmhyn) and wrote it to ./example-folder/QmdQdu1fkaAUokmkfpWrmPHK78F9Eo9K2nnuWuizUjmhyn
+
+-- Going to connect to a few nodes in the Network as bootstrappers --
+Fetching a file from the network with CID QmUaoioqU7bxezBQZkUcgcSyokatMY71sxsALxQmRRrHrj
+Wrote the file to ./example-folder/QmUaoioqU7bxezBQZkUcgcSyokatMY71sxsALxQmRRrHrj
+
+All done! You just finalized your first tutorial on how to use go-ipfs as a library
+```
+
+## Understanding the example
+
+In this example, we add a file and a directory with files; we get them back from IPFS; and then we use the IPFS network to fetch a file that we didn't have in our machines before.
+
+Each section below has links to lines of code in the file [main.go](./main.go). The code itself will have comments explaining what is happening for you.
+
+### The `func main() {}`
+
+The [main function](./main.go#L202-L331) is where the magic starts, and it is the best place to follow the path of what is happening in the tutorial.
+
+### Part 1: Getting an IPFS node running
+
+To get [get a node running](./main.go#L218-L223) as an [ephemeral node](./main.go#L114-L128) (that will cease to exist when the run ends), you will need to:
+
+- [Prepare and set up the plugins](./main.go#L30-L47)
+- [Create an IPFS repo](./main.go#L49-L68)
+- [Construct the IPFS node instance itself](./main.go#L72-L96)
+
+As soon as you construct the IPFS node instance, the node will be running.
+
+### Part 2: Adding a file and a directory to IPFS
+
+- [Prepare the file to be added to IPFS](./main.go#L166-L184))
+- [Add the file to IPFS](./main.go#L240-L243))
+- [Prepare the directory to be added to IPFS](./main.go#L186-L198))
+- [Add the directory to IPFS](./main.go#L252-L255))
+
+### Part 3: Getting the file and directory you added back
+
+- [Get the file back](./main.go#L265-L268))
+- [Write the file to your local filesystem](./main.go#L270-L273))
+- [Get the directory back](./main.go#L277-L280))
+- [Write the directory to your local filesystem](./main.go#L282-L285))
+
+### Part 4: Getting a file from the IPFS network
+
+- [Connect to nodes in the network](./main.go#L293-L310))
+- [Get the file from the network](./main.go#L318-L321))
+- [Write the file to your local filesystem](./main.go#L323-L326))
+
+### Bonus: Spawn a daemon on your existing IPFS repo (on the default path ~/.ipfs)
+
+As a bonus, you can also find lines that show you how to spawn a node over your default path (~/.ipfs) in case you had already started a node there before. To try it:
+
+- [Comment these lines](./main.go#L219-L223))
+- [Uncomment these lines](./main.go#L209-L216))
+
+## Voilá! You are now a go-ipfs hacker
+
+You've learned how to spawn a go-ipfs node using the go-ipfs core API. There are many more [methods to experiment next](https://godoc.org/github.com/ipfs/interface-go-ipfs-core). Happy hacking!

docs/examples/go-ipfs-as-a-library/main.go

+package main
+
+import (
+	"context"
+	"fmt"
+	"io/ioutil"
+	"log"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+
+	config "github.com/ipfs/go-ipfs-config"
+	files "github.com/ipfs/go-ipfs-files"
+	libp2p "github.com/ipfs/go-ipfs/core/node/libp2p"
+	icore "github.com/ipfs/interface-go-ipfs-core"
+	icorepath "github.com/ipfs/interface-go-ipfs-core/path"
+	peerstore "github.com/libp2p/go-libp2p-peerstore"
+	ma "github.com/multiformats/go-multiaddr"
+
+	"github.com/ipfs/go-ipfs/core"
+	"github.com/ipfs/go-ipfs/core/coreapi"
+	"github.com/ipfs/go-ipfs/plugin/loader" // This package is needed so that all the preloaded plugins are loaded automatically
+	"github.com/ipfs/go-ipfs/repo/fsrepo"
+	"github.com/libp2p/go-libp2p-core/peer"
+)
+
+/// ------ Setting up the IPFS Repo
+
+func setupPlugins(externalPluginsPath string) error {
+	// Load any external plugins if available on externalPluginsPath
+	plugins, err := loader.NewPluginLoader(filepath.Join(externalPluginsPath, "plugins"))
+	if err != nil {
+		return fmt.Errorf("error loading plugins: %s", err)
+	}
+
+	// Load preloaded and external plugins
+	if err := plugins.Initialize(); err != nil {
+		return fmt.Errorf("error initializing plugins: %s", err)
+	}
+
+	if err := plugins.Inject(); err != nil {
+		return fmt.Errorf("error initializing plugins: %s", err)
+	}
+
+	return nil
+}
+
+func createTempRepo(ctx context.Context) (string, error) {
+	repoPath, err := ioutil.TempDir("", "ipfs-shell")
+	if err != nil {
+		return "", fmt.Errorf("failed to get temp dir: %s", err)
+	}
+
+	// Create a config with default options and a 2048 bit key
+	cfg, err := config.Init(ioutil.Discard, 2048)
+	if err != nil {
+		return "", err
+	}
+
+	// Create the repo with the config
+	err = fsrepo.Init(repoPath, cfg)
+	if err != nil {
+		return "", fmt.Errorf("failed to init ephemeral node: %s", err)
+	}
+
+	return repoPath, nil
+}
+
+/// ------ Spawning the node
+
+// Creates an IPFS node and returns its coreAPI
+func createNode(ctx context.Context, repoPath string) (icore.CoreAPI, error) {
+	// Open the repo
+	repo, err := fsrepo.Open(repoPath)
+	if err != nil {
+		return nil, err
+	}
+
+	// Construct the node
+
+	nodeOptions := &core.BuildCfg{
+		Online:  true,
+		Routing: libp2p.DHTOption, // This option sets the node to be a full DHT node (both fetching and storing DHT Records)
+		// Routing: libp2p.DHTClientOption, // This option sets the node to be a client DHT node (only fetching records)
+		Repo: repo,
+	}
+
+	node, err := core.NewNode(ctx, nodeOptions)
+	if err != nil {
+		return nil, err
+	}
+
+	// Attach the Core API to the constructed node
+	return coreapi.NewCoreAPI(node)
+}
+
+// Spawns a node on the default repo location, if the repo exists
+func spawnDefault(ctx context.Context) (icore.CoreAPI, error) {
+	defaultPath, err := config.PathRoot()
+	if err != nil {
+		// shouldn't be possible
+		return nil, err
+	}
+
+	if err := setupPlugins(defaultPath); err != nil {
+		return nil, err
+
+	}
+
+	return createNode(ctx, defaultPath)
+}
+
+// Spawns a node to be used just for this run (i.e. creates a tmp repo)
+func spawnEphemeral(ctx context.Context) (icore.CoreAPI, error) {
+	if err := setupPlugins(""); err != nil {
+		return nil, err
+	}
+
+	// Create a Temporary Repo
+	repoPath, err := createTempRepo(ctx)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create temp repo: %s", err)
+	}
+
+	// Spawning an ephemeral IPFS node
+	return createNode(ctx, repoPath)
+}
+
+//
+
+func connectToPeers(ctx context.Context, ipfs icore.CoreAPI, peers []string) error {
+	var wg sync.WaitGroup
+	peerInfos := make(map[peer.ID]*peerstore.PeerInfo, len(peers))
+	for _, addrStr := range peers {
+		addr, err := ma.NewMultiaddr(addrStr)
+		if err != nil {
+			return err
+		}
+		pii, err := peerstore.InfoFromP2pAddr(addr)
+		if err != nil {
+			return err
+		}
+		pi, ok := peerInfos[pii.ID]
+		if !ok {
+			pi = &peerstore.PeerInfo{ID: pii.ID}
+			peerInfos[pi.ID] = pi
+		}
+		pi.Addrs = append(pi.Addrs, pii.Addrs...)
+	}
+
+	wg.Add(len(peerInfos))
+	for _, peerInfo := range peerInfos {
+		go func(peerInfo *peerstore.PeerInfo) {
+			defer wg.Done()
+			err := ipfs.Swarm().Connect(ctx, *peerInfo)
+			if err != nil {
+				log.Printf("failed to connect to %s: %s", peerInfo.ID, err)
+			}
+		}(peerInfo)
+	}
+	wg.Wait()
+	return nil
+}
+
+func getUnixfsFile(path string) (files.File, error) {
+	file, err := os.Open(path)
+	if err != nil {
+		return nil, err
+	}
+	defer file.Close()
+
+	st, err := file.Stat()
+	if err != nil {
+		return nil, err
+	}
+
+	f, err := files.NewReaderPathFile(path, file, st)
+	if err != nil {
+		return nil, err
+	}
+
+	return f, nil
+}
+
+func getUnixfsNode(path string) (files.Node, error) {
+	st, err := os.Stat(path)
+	if err != nil {
+		return nil, err
+	}
+
+	f, err := files.NewSerialFile(path, false, st)
+	if err != nil {
+		return nil, err
+	}
+
+	return f, nil
+}
+
+/// -------
+
+func main() {
+	/// --- Part I: Getting a IPFS node running
+
+	fmt.Println("-- Getting an IPFS node running -- ")
+
+	ctx, _ := context.WithCancel(context.Background())
+
+	/*
+		// Spawn a node using the default path (~/.ipfs), assuming that a repo exists there already
+		fmt.Println("Spawning node on default repo")
+		ipfs, err := spawnDefault(ctx)
+		if err != nil {
+			fmt.Println("No IPFS repo available on the default path")
+		}
+	*/
+
+	// Spawn a node using a temporary path, creating a temporary repo for the run
+	fmt.Println("Spawning node on a temporary repo")
+	ipfs, err := spawnEphemeral(ctx)
+	if err != nil {
+		panic(fmt.Errorf("failed to spawn ephemeral node: %s", err))
+	}
+
+	fmt.Println("IPFS node is running")
+
+	/// --- Part II: Adding a file and a directory to IPFS
+
+	fmt.Println("\n-- Adding and getting back files & directories --")
+
+	inputBasePath := "./example-folder/"
+	inputPathFile := inputBasePath + "ipfs.paper.draft3.pdf"
+	inputPathDirectory := inputBasePath + "test-dir"
+
+	someFile, err := getUnixfsNode(inputPathFile)
+	if err != nil {
+		panic(fmt.Errorf("Could not get File: %s", err))
+	}
+
+	cidFile, err := ipfs.Unixfs().Add(ctx, someFile)
+	if err != nil {
+		panic(fmt.Errorf("Could not add File: %s", err))
+	}
+
+	fmt.Printf("Added file to IPFS with CID %s\n", cidFile.String())
+
+	someDirectory, err := getUnixfsNode(inputPathDirectory)
+	if err != nil {
+		panic(fmt.Errorf("Could not get File: %s", err))
+	}
+
+	cidDirectory, err := ipfs.Unixfs().Add(ctx, someDirectory)
+	if err != nil {
+		panic(fmt.Errorf("Could not add Directory: %s", err))
+	}
+
+	fmt.Printf("Added directory to IPFS with CID %s\n", cidDirectory.String())
+
+	/// --- Part III: Getting the file and directory you added back
+
+	outputBasePath := "./example-folder/"
+	outputPathFile := outputBasePath + strings.Split(cidFile.String(), "/")[2]
+	outputPathDirectory := outputBasePath + strings.Split(cidDirectory.String(), "/")[2]
+
+	rootNodeFile, err := ipfs.Unixfs().Get(ctx, cidFile)
+	if err != nil {
+		panic(fmt.Errorf("Could not get file with CID: %s", err))
+	}
+
+	err = files.WriteTo(rootNodeFile, outputPathFile)
+	if err != nil {
+		panic(fmt.Errorf("Could not write out the fetched CID: %s", err))
+	}
+
+	fmt.Printf("Got file back from IPFS (IPFS path: %s) and wrote it to %s\n", cidFile.String(), outputPathFile)
+
+	rootNodeDirectory, err := ipfs.Unixfs().Get(ctx, cidDirectory)
+	if err != nil {
+		panic(fmt.Errorf("Could not get file with CID: %s", err))
+	}
+
+	err = files.WriteTo(rootNodeDirectory, outputPathDirectory)
+	if err != nil {
+		panic(fmt.Errorf("Could not write out the fetched CID: %s", err))
+	}
+
+	fmt.Printf("Got directory back from IPFS (IPFS path: %s) and wrote it to %s\n", cidDirectory.String(), outputPathDirectory)
+
+	/// --- Part IV: Getting a file from the IPFS Network
+
+	fmt.Println("\n-- Going to connect to a few nodes in the Network as bootstrappers --")
+
+	bootstrapNodes := []string{
+		// IPFS Bootstrapper nodes.
+        "/dnsaddr/bootstrap.libp2p.io/ipfs/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
+        "/dnsaddr/bootstrap.libp2p.io/ipfs/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
+        "/dnsaddr/bootstrap.libp2p.io/ipfs/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
+        "/dnsaddr/bootstrap.libp2p.io/ipfs/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
+
+		// IPFS Cluster Pinning nodes
+		"/ip4/138.201.67.219/tcp/4001/ipfs/QmUd6zHcbkbcs7SMxwLs48qZVX3vpcM8errYS7xEczwRMA",
+		"/ip4/138.201.67.220/tcp/4001/ipfs/QmNSYxZAiJHeLdkBg38roksAR9So7Y5eojks1yjEcUtZ7i",
+		"/ip4/138.201.68.74/tcp/4001/ipfs/QmdnXwLrC8p1ueiq2Qya8joNvk3TVVDAut7PrikmZwubtR",
+		"/ip4/94.130.135.167/tcp/4001/ipfs/QmUEMvxS2e7iDrereVYc5SWPauXPyNwxcy9BXZrC1QTcHE",
+
+		// You can add more nodes here, for example, another IPFS node you might have running locally, mine was:
+		// "/ip4/127.0.0.1/tcp/4010/ipfs/QmZp2fhDLxjYue2RiUvLwT9MWdnbDxam32qYFnGmxZDh5L",
+	}
+
+	go connectToPeers(ctx, ipfs, bootstrapNodes)
+
+	exampleCIDStr := "QmUaoioqU7bxezBQZkUcgcSyokatMY71sxsALxQmRRrHrj"
+
+	fmt.Printf("Fetching a file from the network with CID %s\n", exampleCIDStr)
+	outputPath := outputBasePath + exampleCIDStr
+	testCID := icorepath.New(exampleCIDStr)
+
+	rootNode, err := ipfs.Unixfs().Get(ctx, testCID)
+	if err != nil {
+		panic(fmt.Errorf("Could not get file with CID: %s", err))
+	}
+
+	err = files.WriteTo(rootNode, outputPath)
+	if err != nil {
+		panic(fmt.Errorf("Could not write out the fetched CID: %s", err))
+	}
+
+	fmt.Printf("Wrote the file to %s\n", outputPath)
+
+	fmt.Println("\nAll done! You just finalized your first tutorial on how to use go-ipfs as a library")
+}