Author: Phil Emerson

it took a dumb amount of time for me to find the answer™ to the query:

> i just purchased a domain via Cloudflare (used Google Domains but no thanks SquareSpace)

> i set up email for it via Google Workplace, added gmail DNS records (automatically via Workplace setup where it recognizes you’re using Cloudflare and offers to do it for you)

> and of course in Cloudflare set up a 301 redirection rule via page rules

Why isn’t my domain redirecting then?! (Or resolving at all)

Go into your DNS settings in Cloudflare and add the very important two A records for “@” and “www”

For the IP address, enter 19.0.2.1

Adding the new record(s):

Wtf is that IP address I’m entering? It’s a dummy Reserved IP address as described in this Cloudflare forum thread by Laudian:

I too first saw 19.1.2.3 recommended… it’s not on the dedicated list but (1) i’m sure you’d be fine and (2) not my problem.

I used 19.0.2.1 just to be safe *

* i placed all my trust in the first correct looking answer on the forum post (Thanks Laudian)

I’ve read countless other ‘my tech stack’ blog posts and wanted to make my own.

Productivity

Alfred 4 ($35 one time)

Replaces Apple’s search feature (Spotlight) and so much more. See my post about it here.

Magnet ($2.99 one time)

Allows you to snap your windows around. For example, easily snap two windows side by side taking up 50% of your screen width. Or 3 apps at 33.33% full screen…etc.

Toggl Track (Free plan)

Simple track time and categorization spent between projects.

Coding

Sublime Text 3 ($80 one time)

My preference for a code editor. Loads incredibly quickly and has pretty much everything I could ask for.

iTerm 2 (Free)

Terminal replacement with better design/layout options. Probably has other features but haven’t investigated too much. No learning curve assuming you’re already familiar with Terminal.

ngrok.io ($5 per month)

Allows me to easily tunnel localhost to a publically accessable domain.

Hosting

Heroku for Ruby on Rails projects.

DigitalOcean for everything else.

Domains

Namecheap & Google Domains. Doesn’t matter much, but I avoid GoDaddy.

Email, Chat

G-Suite by Google ($6.00 per month per user)

My go to for any email addresses I need set up with a custom domain.

Slack (Free plan)

Discord (Free plan)

Banking Automation

Astra Finance on iOS (Free)

Allows me to connect multiple bank accounts and set up simple rules for automated transfers between accounts. One of my favorites. For example, are you a freelancer? Create a routine to automatically transfer 30% of each incoming payment in your business checking account to set aside in your savings account for tax purposes.

Passwords

1Password ($3.99 per month)

Beyond worth the price tag and slight learning curve. Can securely access all my passwords across devices and never forgot passwords anymore.

Note taking

Organized folders of .md and .txt files opened with code editor of my choice (Sublime Text 3)

Can quickly open notes using Alfred 4’s search feature as well.

How I outperformed BuiltWith and PublicWWW’s free plans with a couple hours and 30 lines of code.

What if you wanted to find a list of websites that are using a specific piece of technology? Like if you wanted a list of all websites using WordPress? Or in my case, all websites built with Shopify?

You might turn to BuiltWith or PublicWWW. These sites scan the source code of websites to determine the technology the website is built on.

Very useful and cool. Until you see the pricing pages:

Now, let me just say – I get it.

The work they’re doing ain’t easy, and the pricing is justifiable, but since I can code, how about I just build my own for free?

The way I see it, this problem has two components:

1. I need a script that can automatically check, when supplied a website domain, if that website is using Shopify or not.
2. I need a really big list of website domains to run the first script against.
   

Once I have both of those pieces, I can put them together to make my Bootleg Builtwith.

Part 1: Determining if a certain website uses Shopify

Let’s start with the first component: When given a website URL, I need a reliable method to check if that website uses Shopify.

The simplest way to check if a website is using Shopify is to inspect the website source, and look for code that only a Shopify store would use.

Now, I already know of such a snippet that is unique to Shopify stores: It’s ‘trekkie’ – which is what Shopify calls their custom analytics solution:

Cool. Let’s use that as a base point and start coding.

Let’s write a dead simple, 15 line, Ruby script to kick this off:

#~/desktop/is-shopify-store.rb

require 'open-uri'

# Prompt me for a URL 
puts "What URL do you want to check?"
url = gets.chomp

# Open the URL I supplied above
html = open(url).read

# Checks the source code for the Shopify specific code snippet
if html.include? "var trekkie = window.ShopifyAnalytics.lib"
	puts 'This site uses Shopify'
else
	puts 'This site does not use Shopify'
end 
 

Cool. Let’s run it, and try testing it against a couple websites to make sure it’s reliable:

• First test using a Shopify site => https://emerson-code-development.myshopify.com
  • That worked. Let’s try a different Shopify website:
• Second test with another Shopify site => https://allbirds.com
  • Once again, looking good. Now let’s try a website that definitely doesn’t use Shopify.
• Third test with a site that doesn’t use Shopify => https://amazon.com
  • Error!

Ah, yes, I should’ve anticipated this but Amazon blocks my script from inspecting the website source code here.

Now I could get crazy and break out a more sophisticated scraping solution like Watir but remember this is bootleg, so let’s think through that extra work for a second…

So if Shopify stores don’t block my script, can’t I just assume that if my script is blocked, that it’s not a Shopify store? Makes sense to me. Let’s do that:

begin
		html = open(url).read
		if html.include? "var trekkie = window.ShopifyAnalytics.lib"
			puts url + ': This site uses Shopify'
			# It's a match! Let's write it to a CSV
			CSV.open("/Users/emerson/Downloads/results-shopify-stores.csv", "a") do |csv|
			  csv << [url]
			end
		else
			puts url + ': This site does not use Shopify'
		end 
	rescue
		puts url + ': Blocked. Probably does not use Shopify'
	end

Okay. Bingo. Part 1’s solution is decent enough.

Part 2: Get a big list of websites

Onto the next one: I need a massive list of domains to check.

Search engines typically use something called spiders here to find and create directories/lists of webpages. Spiders start at one webpage, follow all the links on that webpage, and then the links on those webpages, and so on until it has mapped all the webpages it can get its hands on.

Now if you wanted to build a Builtwith competitor you’d probably need to do something like that, but this is bootleg baby.

I remember Alexa.com has a list of the top million website domains or whatever. I wonder if they make the list publicly available? Surely that’d give me a lot of high ranking websites to check, and high ranked websites are what we’d prefer here anyway, right?

A quick google search revealed this Github thread with some decent leads, and from that thread I have a downloaded CSV of the ‘top million domains’.

Checking the spreadsheet, it only has 764,166 domains, not 1 million… Good enough!

Let’s start with a couple lines of Ruby code that can use a CSV spreadsheet file as an input, and loop through each domain to perform my Shopify check I outlined in part 1.

Oh yeah… the domain list only includes the URL’s base: Eg bestbuy.com, not https://www.bestbuy.com. That’s okay, let’s just assume most domains are using https:// and prepend it to the start of every URL before we check it:

url = 'https://' + url

Cool. One last thing. If my script does detect a Shopify store, let’s go ahead and save that to a separate CSV to serve as my output, so when my scripts finished, I’m left with just list of websites using Shopify.

if html.include? "var trekkie = window.ShopifyAnalytics.lib"
   puts url + ': This site uses Shopify'
   # It's a match! Let's write it to a separate CSV
   CSV.open("/Users/emerson/Downloads/results-shopify-stores.csv", "w") do |csv|
      csv << [url]
   end
else
...

Okay… One hour later and I barely have gotten through 1000 URLs, and none of those are Shopify sites it seems.

Make sense. If you’re one of the top websites in the world, it’d make sense that you’d be using a custom platform to run it.

Given that, to make this a little bit easier, let’s go ahead and skip the first 50K top sites and start scanning at 50,001:

domainList = CSV.read('/Users/emerson/Downloads/top-1m.csv', headers:true)
domainList.drop(50000).each do |row|
    url = row[1]
    url = 'https://' + url
    isUsingShop(url)
end

A couple minutes later and we got one!

After letting that script run in the background for a few hours, I’ve checked about ~12K domains (Alexa’s top 50K-62K domains), and found ~220 of those sites are using Shopify. So 1.83% of the so far checked domains were using Shopify.

Quick and dirty math: Roughly 700K domains still need to be checked. Assuming similar positive rate as the domains I have checked so far, my bootleg script should produce a list of 10K+ high ranking Shopify stores.

Free BuiltWith: 50 high ranking Shopify stores

Free PublicWWW: 2359 high ranking Shopify stores

30 lines of Ruby code: 10K+ high ranking Shopify stores

Also, here is complete Ruby script:

require 'csv'
require 'open-uri'
def isUsingShop(url)
	begin
		html = open(url).read
		if html.include? "var trekkie = window.ShopifyAnalytics.lib"
			puts url + ': This site uses Shopify'
			# It's a match! Let's write it to a CSV
			CSV.open("/Users/emerson/Downloads/results-shopify-stores.csv", "a") do |csv|
			  csv << [url]
			end
		else
			puts url + ': This site does not use Shopify'
		end 
	rescue
		puts url + ': Blocked. Probably does not use Shopify'
	end
end 
domainList = CSV.read('/Users/emerson/Downloads/top-1m.csv', headers:true)
domainList.drop(50001).each do |row|
    url = row[1]
    url = 'https://' + url
    isUsingShop(url)
end
 

There is no perfect app, or productivity tool, that will solve all your problems.

Alfred for Mac though, is pretty damn cool, and certainly worthy of a mention.

Beware the overwhelm

Start off slow. Alfred has so many features that if you try to use them all out of the gate you will just get overwhelmed.

Okay, then what’s the ONE feature to sell me on Alfred?

The copy/paste clipboard. Do you use ⌘-C and ⌘-V?

Well, with Alfred, when I press ⌘-Shift-V, instead of pasting instantly, I get this popup:

That’s right! Everything I’ve copied to my clipboard in the past X days (You can adjust how long clipboard history is stored for). I can even see what app it was copied from!

It’s easily searchable, and without a doubt my favorite feature of Alfred.

Incredibly useful to use almost as a little note tool – copy everything you might need at will, and access later in the day.

That looks like a neat feature! What else can Alfred do?

A lot of stuff. I (try to) make it a point to only automate tasks that are worth automating. As such, here are all the Alfred features I use daily, besides the clipboard history which I mentioned above.

Note: I unbinded the Apple spotlight search from ⌘+Space and binded it to alfred instead.

Now, whenever I press ⌘+Space, I get the Alfred window:

Within this little text box, here’s what I can do:

Quickly search for and open files (Built in)

Launch apps (Built in)

Calculator (Built in)

Need to run a quick math q? Just type it in

Connect airpods via Bluetooth: (Workflow)
For when my airpods are connected to my phone and I want to connect to computer. Instead of going into my bluetooth settings and clicking connect, I just use this bad boy… One of my favorites.

Give: (Workflow)

This one opens my latest file in my downloads folder. So after I download something in my email or browser, instead of going to dock > opening finder > downloads > clicking file, I just type give and open that shit, saving me precious seconds every year.

Unicode: (Workflow)

Ya know how every blue moon you need to get a unicode character? For example, that little copyright circle (©)? I use to always just Google for it, find the character I wanted, and then copy and paste it elswehere.

With this unicode workflow, I can instantly search for these characters. Makes it a breeze to search for and inject those little bastards.

For example, I just used it above while writing this article to grab that command symbol! (⌘)

The Tools

I use:

• Ruby on Rails w/ shopify_app gem
• Polaris Components for the app’s admin design, but I use the pure CSS/HTML method instead of the intended use as a React component bc I’m a neanderthal
• Ngrok to give my localhost server a secure public URL
• Heroku for hosting the app

The Setup

When you start building apps for Shopify, one thing that isn’t clear is how you should config the apps in the Shopify admin.

I reached out to Ryan Kulp awhile back with this question, and he was kind enough to set me on the right path:

1. Create two Shopify apps in the admin, one for production/app store that points to heroku and one for development that points to ngrok/localhost.

2. Then, in your RoR app within /config/initalizers/shopify_app.rb you want to use your development app keys for development and your production app keys for production (duh).

# config/initalizers/shopify_app.rb
ShopifyApp.configure do |config|
  config.application_name = "App Name"
  if(Rails.env.production?)
   config.api_key = "ProductionKeyHere"
   config.secret = "ProductionSecretHere"
  else 
    config.api_key = "DevelopmentKeyHere"
    config.secret = "DevelopmentSecretHere"
  end 

(You should accomplish this with env variables if you will be putting app code in repository.)

Then it’s as simple as developing in localhost w your development app, and when you’re ready to push those changes in production, run your git push heroku master terminal command

If you’re reading this, you’re probably confused why the helper functions in helpers/application_helper.rb aren’t working within your Shopify app.

My favorite fix for this is to add a small snippet of code to config/application.rb:

config.to_prepare do
   ShopifyApp::AuthenticatedController.helper ShopifyRoRAppName::Application.helpers
end

It is very important to replace the ShopifyRoRAppName with your own RoR app’s name. For reference, it should be right above where you place the code:

So you have an idea for a new business, app, whatever.

Great. If you’re like me, you’d hop on the domain registrar of your choice and figure out the perfect name.

This is a good way to have a pricey portfolio of useless names, as it is incredibly likely (1) it not being such a good idea after all or (2) the solution deviates from your original idea and the name no longer fits.

Do the hard stuff first and the fun stuff after. Easy to say, but not as easy to implement. I constantly need to remind myself of this.