First, I'd like to appologize to everyone for shutting down remotepixel.ca without notice last Friday. This blog post will explain why I had not choice.

Second, I need to re-introduce myself so the following will make more sense. My nane is Vincent Sarago, I'm the `OWNER` (read: `the only person working on it`) of this website. Remotepixel.ca started as PERSONAL PROJECT and will stay a hobby for my cold winter night (winter is quite long in Montreal, QC) and now I have a `real` job that I love and keeps me busy during the week.

Remotepixel is Free... not to everyone

Remotepixel is not just a webpage, almost every project I worked on has one or more services (API) associated. I host Remotepixel.ca web pages and services on Amazon Web Services (AWS) platform and last week when I openned the `billing section` I had this nice surprise ā˜ļø.

The bill was around 3 500 $ (4 510 $ CAD) šŸ˜±šŸ˜±šŸ˜±šŸ˜±šŸ˜±.

My first reaction was to log off because I though I logged with my Mapbox credentials, but then the reallity was brutal, this was my own bill. At this point I had no choice but to shut down everything as fast as I could.

RemotePixel's AWS cost in August

While I'm usually happy to support the cost of running remotepixel.ca (~100$/month), this was 3000% more than my usual bill, and way too high for my `happy to give it for free budget`.

What happened

Remotepixel was finally popular with thousand of thousand of people using it.

ā˜ļø even with this possibility, AWS services are not that expensive, and google analytics quickly anwser this question: RemotePixel is still a `niche` with around 5000 users per month.

RemotePixel's 2018 Google analytics report.

So the web users are not to blame (no huge increate in users or in session duration), but there is something odd the billing is saying.

AWS Lambda eu-central-1 usage in July AWS Lambda eu-central-1 usage in August

RemotePixel's services are based on AWS Lambda which is a service that let you run python/node/java scripts without caring for a server (serverless). I'm running AWS Lambda functions in each region where I need to access data (eu-central-1 for Sentinel-2, us-west-2 for Landsat-8), so it's then easy to see if a specific dataset/services was used more.

In the previous screenshots we can see that AWS lambda function in `eu-central-1` (sentinel-2 access) where called 1.5 MILLION times more in august in comparison to july (which was also an expensive month ~180$). Note how the price for the other region didn't change much !!!

But AWS Lambda pricing is cheap. This cost me `only` 290$. šŸ˜’šŸ˜’šŸ˜’ We still need to explain for 3100 $.

GET/LIST

Remember my post about Sentinel-2 S3 bucket moving to `requester-pays` ? Well this is now in place and we don't have to look too far to see the first consequence. If we look at my S3 cost in eu-central-1 the total is around 3000$ šŸŽ‰.

S3 bill in eu-central-1

480 000 LIST + 950 000 GET request in one month šŸ˜±šŸ˜±šŸ˜±

Is this my fault ? Mostly

Because Sentinel-2 bucket is in `requester-pays` it means that if you have an application accessing the data running outside the `eu-central-1` region, you'll be charged with the data transfer cost. But RemotePixel services run in `eu-central-1` so it should be costless... Well that's not true and the previous screen shot tells you why. You don't pay for data transfert but you have to pay for every LIST/GET requests.

In general, bucket owners pay for all Amazon S3 storage and data transfer costs associated with their bucket. A bucket owner, however, can configure a bucket to be a Requester Pays bucket. With Requester Pays buckets, the requester instead of the bucket owner pays the cost of the request and the data download from the bucket. The bucket owner always pays the cost of storing data. Reuqester-pays doc

This is something I didn't though about, and because some of my services are doing a lot of LIST/GET, event with a normal usage, my AWS bill would have been high. My Search API seems to be a really bad idea now.

Is this my fault ? Mostly Who's scrapping

The numbers don't lie, remotepixel.ca users didn't cause all of the bill increase but someone has been scrapping my services.

Using some companies API is something, but abusing of a free service and/or scrapping is just not fair. Especially when I always try to publish the code of every service and web page I create.

Next

That's said, the GET/LIST numbers seems really odd and I'll need to deep dive on it (another blog post soon).

  • Starting today remotepixel.ca will regain life again. I've updated services to be less expensive (thanks to Sinergise). That's said I had to deprecate some old project which I couldn't update/maintain, please checkout the project page for more info.
  • Sentinel-2 is disabled in viewer.remotepixel.ca.
  • About the august bill, I've reached out AWS and the bill should be forgiven šŸ¤ž.
  • TERM OF SERVICE

    Since I shut down, I've received a lot of support from different individuals or companies, I might not answered everyone but be sure that I've seen it.

    As usual I also received mail from companies complaining about the website/services being down šŸ˜’.

    So let's be clear again, this is a personal project and:

  • if you are making any kind of money using it you should stop right now.
  • You may not use it for any unlawful purpose..
  • You may not use it in any manner that could damage it.