Go: Reading Files

Table of Contents

Following the Learn Go with Tests guide, exploring file input/output operations in Go.

Build a blog post processor that reads markdown files with structured metadata:

Title: My Blog Post
Description: A great post about Go
Tags: go, programming
---
This is the blog post content.

Multiple paragraphs supported.

Goal: Parse these files into Go structs, extracting:

Title, Description, Tags (metadata)
Body content (everything after ---)

File System Abstractions
#

Go 1.16 introduced io/fs with the fs.FS interface for unified file system access. See the official Go 1.16 release notes.

type FS interface {
    Open(name string) (File, error)
}

Benefits
#

Several types implement fs.FS:

embed.FS
zip.Reader
os.DirFS()

Using standard library interfaces like fs.FS, io.Reader, and io.Writer creates loosely coupled, reusable packages.

Instead of:

var posts []blogposts.Post
posts = blogposts.NewPostsFromFS("some-folder")

Better approach:

var posts []blogposts.Post
posts = blogposts.NewPostsFromFS(someFS) // Accept fs.FS interface

Interface Flexibility
#

Using generic interfaces like fs.FS makes arguments swappable. NewPostsFromFS function works with any implementation:

// Production: read from local files
posts := blogposts.NewPostsFromFS(os.DirFS("./posts"))

// Testing: use in-memory files
posts := blogposts.NewPostsFromFS(fstest.MapFS{...})

Read-Only Limitation: The fs.FS interface is read-only by design.

Basic Testing
#

For testing, testing/fstest offers implementations of io/FS, similar to tools in net/http/httptest.

func TestNewBlogPosts(t *testing.T) {
	fs := fstest.MapFS{
		"hello world.md":  {Data: []byte("hi")},
		"hello-world2.md": {Data: []byte("hola")},
	}

	posts := blogposts.NewPostsFromFS(fs)

	if len(posts) != len(fs) {
		t.Errorf("got %d posts, wanted %d posts", len(posts), len(fs))
	}
}

fstest.MapFS: In-memory file system for tests
Benefits: Faster execution, no test file maintenance

External Testing
#

When both blogposts.go and blogposts_test.go are in the same directory, you need to import your own package to test only the external API.

Setup
#

Directory structure:

example.com/tests/
├── blogposts.go      (package blogposts)
└── blogposts_test.go (package blogposts_test)

Test Implementation
#

In blogposts_test.go:

package blogposts_test  // Different package name

import (
    "testing"
    "testing/fstest"
    "example.com/tests" // Import your own module
)

func TestNewBlogPosts(t *testing.T) {
    ...
    // Access via package qualifier
    posts := blogposts.NewPostsFromFS(fs)
    
    ...
}

Package name: Use blogposts_test (with _test suffix)
Import required: Import your own module: "example.com/tests"
Access control: Can only use exported (capitalized) functions
Real consumer: Tests how actual users would use your package, can’t access private functions

Test
#

func TestNewBlogPosts(t *testing.T) {
	const (
		// Sample blog post format with metadata
		firstBody = `Title: Post 1
Description: Description 1
Tags: tdd, go
---
Hello
World`
		secondBody = `Title: Post 2
Description: Description 2
Tags: rust, borrow-checker
---
B
L
M`
	)

	// Create test file system
	fs := fstest.MapFS{
		"hello world.md":  {Data: []byte(firstBody)},
		"hello-world2.md": {Data: []byte(secondBody)},
	}
	
	posts, err := blogposts.NewPostsFromFS(fs)

	if err != nil {
		t.Fatal(err)
	}

	if len(posts) != len(fs) {
		t.Errorf("got %d posts, wanted %d posts", len(posts), len(fs))
	}

	assertPost(t, posts[0], blogposts.Post{
		Title:       "Post 1",
		Description: "Description 1",
		Tags:        []string{"tdd", "go"},
		Body: `Hello
World`,
	})
}

func assertPost(
    t *testing.T, 
	got blogposts.Post, 
    want blogposts.Post
) {
	t.Helper()
	if !reflect.DeepEqual(got, want) {
		t.Errorf("got %+v, want %+v", got, want)
	}
}

Implementation
#

File System Operations
#

Main function that processes the directory:

func NewPostsFromFS(fileSystem fs.FS) ([]Post, error) {
	// Read directory contents
	dir, err := fs.ReadDir(fileSystem, ".")
	if err != nil {
		return nil, err
	}
	
	var posts []Post
	for _, f := range dir {
		// Process each file individually
		post, err := getPost(fileSystem, f.Name())
		if err != nil {
			return nil, err
		}
		posts = append(posts, post)
	}
	return posts, nil
}

fs.ReadDir - Reads directory contents from any fs.FS, returning []DirEntry
DirEntry - Represents directory entries with file information

Helper function to process individual files:

func getPost(fileSystem fs.FS, fileName string) (Post, error) {
	postFile, err := fileSystem.Open(fileName)
	if err != nil {
		return Post{}, err
	}
	defer postFile.Close()
	return newPost(postFile) // Parse file content
}

Post Structure and Constants
#

Define the Post type and parsing constants:

type Post struct {
	Title       string
	Description string
	Tags        []string
	Body        string
}

const (
	titleSeparator       = "Title: "
	descriptionSeparator = "Description: "
	tagsSeparator        = "Tags: "
)

Content Parsing with bufio.Scanner
#

Main parsing function using bufio.Scanner:

func newPost(postBody io.Reader) (Post, error) {
	// Use bufio.Scanner for line-by-line reading
	scanner := bufio.NewScanner(postBody)

	// Helper to read metadata lines
	readMetaLine := func(tagName string) string {
		scanner.Scan()
		return strings.TrimPrefix(scanner.Text(), tagName)
	}

	return Post{
		Title:       readMetaLine(titleSeparator),
		Description: readMetaLine(descriptionSeparator),
		Tags:        strings.Split(readMetaLine(tagsSeparator), ", "),
		Body:        readBody(scanner), // Read remaining content
	}, nil
}

io.Reader interface: newPost accepts io.Reader, so parsing works with any readable source:

Files (from fs.FS)
Strings (strings.Reader)
In-memory data (bytes.Reader)

Body content extraction:

func readBody(scanner *bufio.Scanner) string {
	scanner.Scan() // Skip separator line (---)
	buf := bytes.Buffer{}
	for scanner.Scan() {
		fmt.Fprintln(&buf, scanner.Text())
	}
	return strings.TrimSuffix(buf.String(), "\n")
}

scanner.Scan() - Advances to next line, returns false at EOF
scanner.Text() - Returns current line as string

Usage
#

Real-world usage with local file system:

func main() {
	// Use os.DirFS to read from local directory
	posts, err := blogposts.NewPostsFromFS(os.DirFS("posts"))
	
	if err != nil {
		log.Fatal(err)
	}
	log.Println(posts)
}

os.DirFS("posts") - Creates fs.FS from local directory
Same interface - Works identically to test fstest.MapFS

File System Abstractions #

Benefits #

Interface Flexibility #

Basic Testing #

External Testing #

Setup #

Test Implementation #

Test #

Implementation #

File System Operations #

Post Structure and Constants #

Content Parsing with bufio.Scanner #

Usage #