Go and Oauth2: Part 1

February 3, 2024

background

As much as you can’t avoid IoT devices - especially in home automation - the one thing that seems to suck about all of the devices is telemetry. Specifically, metrics over time. Some devices have maybe 7 days of data with a few graphs but if you have multiple devices of different brands it’s basically impossible to see the data collated in one place, ya know to make it useful?

Also I’m pretty dumb.I write about dumb things so that other fellow dummies can hopefully learn from my dumbness.

This is the journey of wrangling Google APIs, Nest APIs, Google OAuth2, and somehow GCP???

goal

My intention was to build something that would read data from my Nest Thermostat on a decent interval( like 5 mins ) and export that data to a TSDB like InfluxDB. That way I could create graphs in grafana and also cross reference other metrics like CO2, pressure changes, etc. Why? Well over the winter in the midwest, you use the heat/furnace a lot. I had a suspicion that we didn’t have enough airflow which was made worse when the furnace would turn on. Maybe the fan when blowing doesn’t circulate or something? No clue - but can’t really start diagnosing until we have some data to look at. That’s a long winded way to say that the Nest could expose events as well like when the heat is turned on and off so that those events could show up on a CO2 graph like deployments on an API.

research

Doing some looking around to find blogs, packages, whatever to see if anyone really has gone down this path. The closest thing I found was this. It’s a really good start going through the setup and auth bits but was really lost on why there was a portion of the authorize() function that is used in the OAuth Web Flow to return the code in the query params in your browser back to the app running using r.URL.Query().Get("code"). This then is passed into token, err := config.Exchange(ctx, code) to return a valid OAuth2 Token to use in making calls to the Smart Device Management API for Nest Devices. I was stuck on how this was going to run on a server( or NUC ) in something like a crontab every 5 minutes if I had to go through the OAuth Web Flow every time it needed to connect because it would need a new token. There was one line in that blog that pointed to a clue:

“After config.Exchange(ctx, code) you should cache this and use it in future instead of using the web flow every time.”

Now, again referencing above - I’m pretty dumb but I have built auth systems for companies with 10’s of thousands of users, 100’s of apps, spanning globally. I had no clue WTF this meant. I mean on a browser side this would look something like this:

Login to web app -> forwards login request to SSO/IdP -> redirects to app with token -> token is stored as cookie.

This way when your token expiry times out the cookie is refreshed usually hitting an endpoint for your IdP, asking for more scopes to do things in the app, etc. BUT to do this, you aren’t logged out of the app - typically.

NOTE: There’s typically 2 timeouts or expirations. 1 for your token which is set when it’s created. Usually an hour or something short. Then your login session from the IdP something like 8 hours. What this means is as your login session is valid your not logged out of all your apps when your token expires, it just renews the token.

Back from that aside, my best guess is I have to store this token somehow, somewhere? I guess it makes sense when thinking through traditional models of how tokens are passed around to API backends.

Frontend takes token -> passes to API backend request as Bearer Header -> backend takes token, validates scopes or gets more if needed -> responds back to frontend with response.

Nowhere in there is the token stored for refresh later, the frontend system that is communicated with the API handles that. Even if it is expired, the backend can attempt to get a refresh token to continue the request.

This is all a super long winded way to say: WTF do I do with this token? Remember, I’m using Go so I can create a binary that runs in a crontab and spits out to InfluxDB. Where is this token going to live? Even running a SQLiteDB seems excessive.

Here’s where things get wild - Obviously, we take to our favorite search engine and this Open Issue comes up for the oauth2 pkg. Best part is this has been open since Jan 2015! Now, if you go read it, and there’s a lot there, it will eventually make some sense. My TL:DR is this:

TL:DR There used to be a function that would cache the token as a file. They removed because maybe people wanted to handle the token themselves like in a db or something. This makes sense as an open source project that is trying to implement a RFC. Where it get’s confusing - or so it seems for many people - is how to handle this now that there’s no helper function.

This almost 10 year old issue could probably be solved with an example somewhere in the repo, pinned in the issue. Idk, something that points the large amount of people to an answer on how to solve this. In there its just links to random PR’s or copy pastas of how other have solved it in half pseudo code.

So here, I’ll walk through in excruciating detail how this damn thing works so that if someone has this issue and stumbles across it they have a working solution that they can use to at least get them moving.

prerequisites

We need 3 things before we can start

OAuth Client ID
OAuth Client Secret
Nest Project ID

OAuth Client ID/Secret

Create a new project in Google Developer Console. Then enable Smart Device Management. When done in OAuth consent screen tab, click edit. Add your user to the Test Users - this is so your user can authorize to your Nest API with the correct scopes. When done, under the Credentials tab. Create a new Client ID for Web Applications. Set the Authorized redirect URIs to http://localhost:8080 for now. Copy the ClientID/ClientSecret for later use.

Nest Project ID

This part is kind of lame. Go to the devices access and unfortunately pay $5. Once it finishes, go to projects. Create a new project and the only important part is to add the Client ID from earlier into the creation screen.

code

I will put snippets here, but here’s a link to the completed project to view it in its entirety. If it wasn’t clear yet, this project is all in Golang. If you’re not used to using go mod, links here.

First Part - OAuth2 Client

main.go

import (
	"github.com/joho/godotenv"
	"golang.org/x/oauth2"
	"golang.org/x/oauth2/google"
)

func main() {
	err := godotenv.Load()
	if err != nil {
		log.Fatal("error loading .env file")
	}

	clientID := os.Getenv("CLIENT_ID")
	clientSecret := os.Getenv("CLIENT_SECRET")
	projectID := os.Getenv("PROJECT_ID")

	ctx := context.Background()
	conf := &oauth2.Config{
		ClientID:     clientID,
		ClientSecret: clientSecret,
		RedirectURL:  "http://localhost:8080",
		Scopes: []string{
			smartdevicemanagement.SdmServiceScope,
		},
		Endpoint: oauth2.Endpoint{
			AuthURL:  fmt.Sprintf("https://nestservices.google.com/partnerconnections/%s/auth", projectID),
			TokenURL: google.Endpoint.TokenURL,
		},
	}

.env

CLIENT_ID='your client id'
CLIENT_SECRET='your client secret'
PROJECT_ID='your project id'

Here I’m using godotenv which will automatically load your .env file as environment variables that you can load in. Also don’t take this as production ready code, this is for explanations.

We set a context to use later for making our auth calls. The meat of this snippet is the oauth2.Config we initialize. For the Nest this can be copy pasted. The important bits here are the scopes and the redirect url matching what you set in the Nest Project settings.

Second Part - File Helper Functions

The first 2 we will create are for reading the token from file and saving our token to file.

func SaveTokenToFile(oAuthToken *oauth2.Token) {
	file, err := os.Create("token.json")
	if err != nil {
		log.Fatal(err)
	}
	defer file.Close()

	p, err := json.Marshal(oAuthToken)
	if err != nil {
		log.Fatal(err)
	}

	_, err = file.Write(p)
	if err != nil {
		log.Fatal(err)
	}
}

func ReadTokenFromFile() *oauth2.Token {
	file, err := os.ReadFile("token.json")
	if err != nil {
		log.Fatal(err)
	}

	var t *oauth2.Token

	err = json.Unmarshal(file, &t)
	if err != nil {
		log.Fatal(err)
	}

	return t
}

SaveTokenToFile() is pretty straight forward. It takes an *oauth2.Token, creates a file called token.json, marshals the token to json, then writes it to the file.

ReadTokenFromFile() does the opposite of the save function. It reads from the token.json file, unmarshals it from json into an *oauth2.token. Again, pretty simple helpers.

The last bit brings it all together in maybe not the most elegant way but works here to provide a better way to show conceptually what is happening.


var token *oauth2.Token

if _, err := os.Stat("token.json"); errors.Is(err, os.ErrNotExist) {
	token = authenticate(ctx, conf)
}

token = ReadTokenFromFile()
if token.Expiry.Before(time.Now()) {
	src := conf.TokenSource(ctx, token)
	newToken, err := src.Token()
	if err != nil {
		log.Fatal(err)
	}
	if newToken.AccessToken != token.AccessToken {
		SaveTokenToFile(newToken)
		token = newToken
	}
}

This looks dumb but the concept is simple. Check for a file, if its not there we run the authenticate() function - more on that below - but it will create the initial token. If the file exists, we read it, check if its expired, if it is, then refresh it. The part here that is confusing is this line:

src := conf.TokenSource(ctx, token)

It’s not immediately clear but that takes the “expired” token and uses it to create a new token with the client that doesn’t require the OAuth2 web flow action(WHAT WE WANT).

From there, we now have a way to get an initial token and refresh the token without a full web auth flow. We have one more piece to go over before we can move to part 2 of getting the nest metrics.

The last piece to explain is the authenticate() function. That looks like this:

func authenticate(ctx context.Context, config *oauth2.Config) *oauth2.Token {
	log.Print("Your browser will be opened to authenticate with Google")
	log.Print("Hit enter to confirm and continue")
	url := config.AuthCodeURL("state", oauth2.AccessTypeOffline)
	codeChannel := make(chan string)
	mux := http.NewServeMux()
	server := http.Server{Addr: ":8080", Handler: mux}
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("<p style=font-size:xx-large;text-align:center>return to your terminal</p>"))
		codeChannel <- r.URL.Query().Get("code")
	})
	openBrowser(url)
	go func() {
		if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Fatal(err)
		}
	}()
	code := <-codeChannel
	log.Print("Shutting down server")
	server.Shutdown(context.Background())
	log.Print("Exchanging token")
	log.Print("Code: ", code)
	token, err := config.Exchange(ctx, code)
	if err != nil {
		log.Fatal(err)
	}
	return token
}

The function takes a context and *oauth2.Config returning a *oauth2.Token, makes sense. After the log prints we call config.AuthCodeURL() which returns the url that we will display to login with our google account and authorize the scopes the app will use for us. These are those screens you’ve seen millions of times when doing the “Login with Google” options on websites.

The tricky bit here is where the channel is created. This is so the response code returned from the config.AuthCodeURL() flow can be passed down to the config.Exchange() function. The mux server parts are pretty straight forward in creating a listener to handle the callback from Google Auth Flow - the address is what we configured above in the Authorized Redirect URI screen. You can set this to whatever, they just have to match. More on that part later.

So now we have a way to push our auth url and build a endpoint for the callback to hit. The next part is this openBrowser() function. That was referenced in this but originally points to a gist here. It looks like this:

func openBrowser(url string) {
	var err error

	switch runtime.GOOS {
	case "linux":
		err = exec.Command("xdg-open", url).Start()
	case "windows":
		err = exec.Command("rundll32", "url.dll,FileProtocolHandler", url).Start()
	case "darwin":
		err = exec.Command("open", url).Start()
	default:
		err = fmt.Errorf("unsupported platform")
	}
	if err != nil {
		log.Fatal(err)
	}
}

A nice helper that figures out what OS you’re on and then opens the URL we created in the browser.

We have an auth url, http callback endpoint, and a way to open that url in the browser. Now we need to grab the “code” created from the auth session which if you try by hand or check the URL in your browser you will see it as a query param. We use that code in the config.Exchange() function that returns a token. Technically, this takes a authorization code ( code ) and returns a token.

And that’s it. We have a valid token we can now use to create a client for making SDK calls.

Takeaways

This was a bit of an adventure. If the smartdevicemanagement API allowed use of an API key, this would have been so much easier. I think the issue falls into a simple explanation.

Because we are using Google’s Auth which in turn is actually GCP to authorize into Nest which is also Google Auth but not in a way we have access - unless we are a device partner or something.

An example would be we have a company with our own auth system and IDP. We want to allow users to add their nest to our home automation platform. We configure our Nest Project to use this systems client id/secret to connect to the nest platform which in turn starts the whole OAuth web flow for them. Once initially connected, we have a token saved for them in our DB for when we need to interact with the Nest APIs.

I think this is where a lot of confusion comes in. Everyone has vastly different needs for their integrations and smartly, the oauth2 package doesn’t force a direction. Where it falls short is explaining this and giving some useful examples.

next steps

We got our auth working now - even if only setup for a local machine. Now it’s time to add some code for connecting to the Nest API and pulling some data. Maybe then we can finally get towards pushing that to InfluxDB!

donald.dev