In this episode...

• What is AirCare, and what does it do for you?
• What is its business model?
• AirCare's origin story
• AirCare's tech stack: Flutter, PHP, MongoDB, DigitalOcian
• Request volume and seasonality
• How to aggregate 35,000 data sources in PHP
• Detecting and responding to a DoS attack
• The software development cycle at AirCare
• Observability at AirCare
• Scaling challenges along the way
• Next steps and objectives

Guest
Gorjan Jovanoski
AirCare web site

Transcript

Announcer: Ladies and gentlemen, the Tiny DevOps guy.

[music]

Jonathan Hall: Hello, everybody. Welcome to another episode of the Tiny DevOps Podcast. I'm your host, Jonathan Hall. Today with me, I have, I'm going to try to say this right, Gorjan Jovanovski. Did I get it close?

Gorjan Jovanovski: Wonderful. You got it. You got it on spot.

Jonathan: Thanks for coming on today, Gorjan. We're going to be talking about your company, AirCare. Is that right? I don't know what they do yet. This is going to be just as exciting for me as for everybody listening. Before we talk about the company, why don't you introduce yourself? Tell us how long you've been doing tech, what you do.

Gorjan: First of all, thanks for having me on the show. I'm super happy to share the story. I am a software engineer by trait and an eco-activist by heart. I have been trying to use technology to improve the situation, especially with preserving the environment and battling the climate crisis that we have, not only here in my home country of Macedonia, but also all around the world, wherever I can reach. From a technology perspective, I started teaching myself how to code when I was in 7th grade, way back in elementary school. We had a magazine here about coding, which came out. Unfortunately, I missed the first issue, and I had the second one.

In the second one, they were already talking about, "Hey, look at this code and write it in, and it'll do these things," which was great, but I didn't know where to put the code because I missed the first issue, which had the little CD with the program. It was Visual Basic 6, I think, back in the day. I had to figure all of these things out on the go, use a little magnifying glass to look at the screenshots on the magazine to see what it says on the bar to understand what app it was. Yes, from that point on, I taught myself how to code, and ever since, I've just loved to tinker with technology.

Jonathan: Let's talk about your company because, of course, on this podcast, we like to talk about solving big problems with small teams, small companies. I know you're working on a small company. AirCare is the company. What is the product or service that AirCare sells or is intending to sell?

Gorjan: We want to help you know what you breathe.

Jonathan: Okay.

Gorjan: If you open your window, let's say here, I have a window in front of me. I open this window. I want to know if the air that I'm going to get in my house is going to be good or not. Should I take my dog for a walk or might there be smoke in the air from wildfires, from factories, from burning wood? Basically, helping everybody know what they breathe and then taking it to an extra level apart from that.

Jonathan: I think a lot of people, especially people with allergies, are already familiar with websites. Depending on what country they live in, they can tell you what's the air quality today today. Should I go outside? Should I wear a mask? Whatever. This is going to be more detailed, I guess, than that and maybe more localized. What's the difference?

Gorjan: There's a couple of things. First of all, we try to gather as much data as possible. We have data from, I think last I checked, over 35,000 sensors around the world that we're able to get data for. Now, we're expanding that network as well. This data comes from government, measuring stations that, I don't know, the environmental protection agency, for example, would put out there or my local government here. It comes from volunteer measuring stations, which people like you and I can buy and set up on our windowsill. It also comes from satellite data.

We work with data from the European Space Agency, which have satellites that orbit the earth and record air pollution on the ground. We try to fill in voids where there's no ground data using all of the satellite data. All in all, we want to have a very clear picture, either down to the block level of a city or as wide as a city if there's no sensors there on what you guys breathe. The second part that makes AirCare much more interesting is that we're built on three pillars. Our mission technically is built on three pillars.

We want to inform, we want to educate, and we want to empower. All of the other competitors, all the other websites, will tell you what the air quality is outside, will tell you and they'll tell you. Are we more precise? Do we have more data? Do they have more data? It depends on reach. We don't want to stop there. Great, you know outside is polluted. Now what? What is my next step? We write a lot of content in the app, educational content, to help you know exactly why air pollution is important because unlike, let's say, COVID, for example, air pollution takes a very long time to take a toll on your health. When it does, it's too late to do anything about it.

We want to educate people that, yes, you might go outside and not feel that bad, but this in the long run is going to hurt you. It's going to hurt your family, your pets, the environment. Then the last step is, okay, I know that it's bad outside. I know how it's going to harm me. What do I do about it? This is where we want to engage our users. We want to get them out there to do something for us, either by connecting them to local NGOs, eco-organizations that have workshops, that have protests, or simply to tell them where to lobby, who to talk to, to your local representatives to get more greener and cleaner policies out there. It's not only about informing, but also educating and empowering our users.

Jonathan: Wow. I have a lot of questions here, so this is going to be interesting. They aren't even technical yet, which is I definitely want to get to. [laughs]

Gorjan: Not a problem. Let's do them all.

Jonathan: Let's start at the business level. What's your revenue model? I mean, how do you turn this into something that pays the bills?

Gorjan: Right. We have two revenue streams, B2C, B2B. The B2C side is we have a pro version of the app, which unlocks extra features, basically quality-of-life improvements for people who want to support the team. You can subscribe to it at, I think, $125 a month, which is nothing in today's terms of subscription models. Something to support the team of what we do. The B2B side is, of course, the bigger and more interesting side is where we basically collect all of this air pollution data and then we help companies make better decisions using this data.

Let me give you a clear example, because when I started out with this idea, I said, "Who would want to use air pollution data? What company would care to do this?" First of all, we found out governments care. Governments that care about air quality care about the data as well. There's governments that simply don't care at all, but the ones that do want to know how bad the air quality is outside and want to know if the methods that they're using to better the environment is actually doing anything if you look at the data.

Companies like L'Oreal, companies that develop skincare products, they want to know which markets have problems with air pollution and when, because they have special creams, for example, like on your face, that help with tackling the effects that air pollution has on your skin. There's a lot of these weird use cases that we found out through years of research that companies use this data for.

We sell air pollution data to companies. What we also do is we help them with custom solutions directly on the app. For example, if you are a eco company or a green company or do something to better the environment, you can have ad space directly on the app, and you can reach out to all of our users because that's your very, very targeted audience there. You don't have to do all Facebook and Google ads. You can really target the people who care about the environment.

Jonathan: I don't know, you've got my mind going about activism and environmentalism and all this stuff. Let's touch on that since that's the topic of your company before we dive into the technical stuff.

Gorjan: Sure.

Jonathan: You talked about things we can do as a consumer of the app, and you talked about contacting government representatives and so on. On a day-to-day basis, what do you encourage your users to do? I hope there's more nuance than just don't go outside today because the air is bad. [laughs]

Gorjan: Of course. No, no, no. That's the first step you should do if you go outside, we recommend you wear a mask. Unfortunately, because of COVID, we all got used to wearing these filtered masks, and those actually stop air pollution particles as well. Of course, there's little things you can do there. You can use your bike or use public transport instead of using a car, especially in bigger cities where you have the option to do that. You can recycle, for example, so that not all of your trash goes into one place, which just gets burnt and then again pollutes the air.

If you heat your home using wood, you might want to switch to electricity or use solar energy to do that instead of using wood and again polluting the environment. The whole story of AirCare and where it started, and this is why I talk about lobbying with your elected officials, is because that's how we started, and that's how we made a change in my home country of Macedonia. Because when I was growing up, there's this very incredible picture that's taken by a drone over the city.

You can see that there's what appears to be a city there, but because the smog is so dense, you can only see three skyscrapers that are poking through the smog and then one big mountain on which we have in the year 2000, we built a big cross that lights up a Millennium Cross. Us from Skopje, we tend to have some dark humor and say, "Hey, look, they put a big cross. It's like a graveyard. They buried the city in the pollution, and they put a big cross on top." [laughs] Sometimes we have to use a comedy to write in the situation. Yes, it was quite bad here. Nobody was talking about how bad the air pollution is. Data existed. There were 10-ish sensors from the government installed, but they were being presented on a website that seems to be built in the 90s and forgotten about. If you either had the luck to find them, you wouldn't understand the data that's there. Once I made AirCare into an app that anyone from their own pocket, from their own hand, can see what the air quality is outside using already existing data, mind you, I did not do anything else with no extra data, a lot of people not only got aware about it, but they went out on the streets.

We had massive protests all around the country. Cities were blocked by these protests, I think five of the major cities were, and people started demanding that the government takes serious action because we all breathe the same air at the end of the day. Nothing differentiates you from me, from my neighbor, or from the guy who's pay millions and millions. We all breathe the same air. This is how the whole story got started.

Jonathan: That's a great segue because I wanted to ask about how the company got started. Whose idea was it? Was it yours? I know you work with other people, just tell me that story. Where did the idea come from and how did you get started?

Gorjan: This was during my university years here. I was studying in the local IT university. Together with friends, we were trying to teach ourselves how to code mobile apps. We didn't have a course back then on mobile apps so we started going on YouTube looking at tutorials. I think it was Android back then, I didn't even have an iPhone, so I couldn't test and work with iPhones. It was purely Android-based Java coding. While I was developing my first prototype app, I said, I want something useful.

At least if I'm going to teach myself how to make an app, let's make something that's a bit useful to me. Not just to say hello world, my name is Gorjan, let's do something interesting. This is when I stumbled upon this old website that I mentioned from the Ministry of Environment here. I looked at the website and I don't understand the data. I decide, okay, let's check some endpoints. I did a bit of network request looking up and seeing what things are going on in the background and I found an API.

I said, "Okay, let's download this data, let's check out what it's trying to tell me." I was shocked when my algorithm, which was supposed to give me a final number of the data, constantly kept telling me it's polluted 20 times over EU limits. I constantly kept looking for the bug maybe two times. It can't be 20, maybe 2 times, maybe 0.2 times. No, it was 20 times over EU limits. This is where the shock factor set in and I said, "Wait, you mean that what we're breathing outside is basically cancer? That's what we're breathing? This is not the smell of winter. This is actually smog."

We used to romanticize it. Oh, it smells like winter. No, it's smog, it's crazy. Don't breathe that. I think this is when me and a couple of friends, we said, can we do something about this? This is when AirCare was born back in its old name and its old form. It used to be called Moj Vozduh, which in Macedonian means my air, but we had to rebrand it for global purposes of course. It got started as a student project and it went on to be a student project for quite a long time. I started my activism at that point, 8 years ago.

I decided, hey, this is helping out. A lot of people got interested, a lot of media picked up on the story. I can find some very old cringe interviews of me on local media talking here and being very confused as to how to explain this issue. From then on, it just kept growing and growing and growing locally until one day we decided, hey, you know what? We're not the only ones that face this problem, let's see if other countries also have this issue.

Jonathan: How many of you were there initially when you started the project?

Gorjan: Initially, it was me that was working fully dedicated on the project, but I also had support from three or four people in different areas which I needed. When we decided to develop an iOS app, we had a friend that knew iOS to come along and write it in Swift. Sometimes when I didn't know what I was doing with a server, because that's the first time I ever touched a server back and front end, I was learning on the go. I'd have people helping out. I always say that AirCare, while maybe I was the person that was doing most of the coding, it was always a community effort that built the app.

Jonathan: How many are at the company now?

Gorjan: We are five people now.

Jonathan: Five people. What's the breakdown? Are they all technical or do you have sales and marketing, or what kind of breakdown do you have there?

Gorjan: We have marketing, we have design, we have two people that are technical and we have a copywriter that works on the majority of the content that you see within the app and with our blog.

Jonathan: You're one of the technical people?

Gorjan: Yes.

Jonathan: You [crosstalk]

Gorjan: Unfortunately, or unfortunately, I'm one of the technical people.

[laughter]

Jonathan: All right. Let's talk a little bit. Let's start to get to the nitty gritty about the technical staff, the staff that people are probably here to listen to. What's your text stack like? You've already mentioned Android and iOS, so I think that's probably fairly straightforward. If it's not, then of course feel free to elaborate. What's your back-end like? Where's it hosted? What language are you writing stuff in? Things like that.

Gorjan: Actually, we don't write native code on Android and iOS. We used to have that until three or four years ago? I think probably three years ago when I went to a Google conference in Florida and I discovered what Flutter was. Flutter for maybe those of the listeners who haven't heard of this, is a framework that is developed by Google, which is similar to React Native from Facebook. You write code once, and then it compiles to Android, it compiles to iOS, and lately, you can compile from the web, you can compile to Windows, Linux, Mac.

You can basically compile your app everywhere, which saves us a lot of headache because I never sat down to learn Swift correctly, and I did not want to do it. This way, we have one code base, that means all of our users get the same features at the same time, same experience, instead of having to rely on two different code bases and possibility of bugs and so on and so forth. From a front-end perspective, we work with Flutter. Even the new web version of the app, which is still not publicly available, is also built in Flutter. As for the back-end, we write in the most popular back-end language called PHP, that everybody loves so much.

PHP has helped us prototype very fast. I know a lot of people hate the language, but it's stuck around. Its kind of legacy at this point, which is weird to stay in the startup, but it's stuck around and we have agreed to just keep using it because it works. We have back-end, yes PHP, we work with both MySQL and Mongo databases depending on what type of data we need to store, because we have a lot of data coming in from all of these sensors. All of this is being hosted on the cloud. We use DigitalOcean servers right now, both for the databases and for the back-end.

Jonathan: If you're willing to share this, what kind of request volume do you have these days?

Gorjan: We have a lot. According to Cloudflare, I think the last report that I saw from Cloudflare, they have cashed approximately, I think two or three terabytes of data that went through the service that were supposed to hit our servers. It really depends on season. Right now, the app has two seasonalities. During the summertime when you have fire season in the US, we have a lot of users, especially from California.

During the wintertime, we have a lot of users from Eastern Europe because this is where you have a lot of pollution during the winter, unfortunately, one of the countries where I also come from. It really depends, but we have anywhere from 10 to 200,000 active users per day. It very much varies depending on season, and that's why we tend to make big changes to the server when it's low season, when we can mess up less, let's say.

Jonathan: [chuckles] How many data sources do you have? I think you may have mentioned a number earlier and it was quite a large number.

Gorjan: Around 35,000 data sources.

Jonathan: How are you, I don't know, probing that data? How are you aggregating that? What does that look like?

Gorjan: It's a pull model basically. We have the APIs of all these endpoints. For some of them, it's public APIs, which is great for others, we have to be a bit more creative. For example, especially when it comes down to government and websites, they tend to not like the concept of an API. Sometimes it may come down to scraping a HDML table in order to get the latest data, but we have all of these different providers built-in as basically plug-and-play providers in our main architecture. When you find a new provider-- I don't know.

I was talking to the UNDP mission in Kyrgyzstan. They called me up and they said, "Hey, listen, we heard about your app, but there's no data for Kyrgyzstan. We have a portal which shows it, can it be in your app?" I think it took us less than 30 minutes and I told them, open the app again, and Kyrgyzstan was there. We have built this plug-and-play architecture which basically you just write a little bit of code about the concrete API that you're pinging from, and then there's a clutter up that takes over, and in every hour it would ping this place, it would grab all the data, it would generalize it in our own format, stored in our database, cash it, and then deliver it to users.

Jonathan: You have 35,000 sources that you're querying on a cron job. That's at least 35,000 requests per hour that you're making.

Gorjan: Not really. The good thing is that depending on the APIs-- I really love APIs when you can ping one endpoint and you will get all of the data for all of these stations. It is a huge JSON file, for example, I know it takes a lot of memory, but it makes our life a lot easier so we don't have to do 35,000 requests per hour to these. Some providers don't have that and we have to default to being in every single station every single hour, but we try to generalize this as much as possible so we don't--

Of course, we don't seem like bots at the end of the day, that there are pinging this website and we don't overload their servers as well because a lot of these government servers, including the one from my country, are hosted on a potato, so if you have more than one consecutive users, they will fail. We are kind of even helping out governments in certain countries by saving their infrastructure because people use our app instead of their website, so at least it doesn't crash.

Jonathan: What kinds of data are you gathering? Do you get data about which pollutants are in the air or just how much particulate matter or what?

Gorjan: We try to get as much raw data as possible, so we need the raw data. We take particulate matter, especially PM10, PM2.5. We work with ozone, nitrogen dioxide, sulfur dioxide, carbon monoxide. Basically, anything that a station can provide, we will take, and will show it in the app. The more data the station can provide, the better and clear the picture of air quality is in one place. After we have all of this data, we have to generalize this information. We have to aggregate it into an index. A lot of countries have, unfortunately, different indexes. There is no one global index that everybody agreed to. There is a US version of the Air Quality Index, there is a European one.

There is a Chinese one. There is so many. Right now, we work with both European and US air quality index measurements. Once we take all of this raw data, we use different formulas to calculate this one single number, this one air quality index measurement, and then that's the one that initially anybody in the app can see. Then you can dive deep into the data, into the details and see, okay, it is being derived from these raw value or these raw pollutants, and see what is actually making the problem in the air, is it PM10, or is it ozone, for example?

Jonathan: At a quick glance, a user can just sort of get a number and then they have an idea of what it's like. If they care about the details, then they have that available to the extent that it is available.

Gorjan: Exactly. Exactly. Our idea is to make it simple. We want you with a quick glance, you open the app, even if you don't even look at the number, there's a big colored box, green, yellow, red, purple. Based on the color, you can say, "Okay, I know the air quality outside here is this, that's great," but if you want to dive into more details, you have a bunch of stuff. First of all, you can look at historical graphs.

You can look at predictions based on the satellite data. You can look at the raw pollutants. You can look at a map of every single station on the world that we have data from right now. You can see the predictions for the air quality in places that we don't using the satellite data. You can favor at a bunch of stations and see how it is at my mom's place, for example, or at my my work, or at my school where my children go, for example. You can have kind of a very big overview of what's going on in your area.

Jonathan: You mentioned predictions. I'm curious about all sorts of things there. What kinds of predictions can you make, and I guess the second question is how accurate can they be?

Gorjan: Right. That is a very tricky thing to do. We have tried here, over here, model over model to figure out predictions. What we're doing right now is we actually left the predictions to the experts. These predictions are, again, from the European Space Agency. They have a bunch of models that they run from weather to data, and they just pump out numbers and we say, "Okay, you know what? These are the predictions that the European Space Agency says might happen in the next 72 hours." They are, on the majority of locations, quite good, quite good.

We can see that they overlap, let's say 80% of the time, which is quite okay. Unfortunately, not everywhere can this work. For example, here in the city where I live in, companies tend to get smart and turn off their filters or turn on their incinerators around night, midnight when there's no inspectors out so that they can't get fined. Unfortunately, this is hard to predict. You don't know when these things will happen or will not happen. You can say per season on general in January, it's going to be worse than in June, but you cannot predict it that well.

That's why we kind of keep the predictions as a nice-to-have thing, it's there, but we give a big warning saying, "Hey, listen, these are saddle estimates. This is as far as we've gotten with this." Hopefully, in the future, we'll be able to derive better models, get better predictions. We're working with the local universities here who already have an idea of how they think we can use this. Hopefully, within, I don't know, half a year, we might be able to put their model to the test.

Jonathan: Fascinating. How far into the future do these predictions typically go? Can you make a plan for the next weekend or is it just like tomorrow and the next day or something like that, kind of like--- [crosstalk]?

Gorjan: They go up to 72 hours, so they do have a 72 hour forecast window from the moment that you're querying. That's as far as they go. You can probably find predictions that go up to probably 10 days, I think, but then the quality of those predictions is quite low so we just don't show.

Jonathan: Yes, makes sense. How did you make the decision to go with PHP and with DigitalOcean? What alternatives did you consider and how did you end up there?

Gorjan: For PHP, it was basically what did I know at the time of development. I remember that as a kid, I was actually taking a course, local course here for web development, and they were teaching PHP. That was the one back-end language that I knew back in the day. I said, "Hey, look, it helps me quickly develop, it helps me quickly prototype these things."

As a startup, you tend to want to move fast. Instead of sitting here and doing some big boilerplate framework of things, we are able to quickly develop new features and put them out.

Of course, it comes with a lot of the drawbacks that PHP comes with as a loosely typed language, but it still gets the job done right now. At some point in the future, as we get more tech people on board, we might be migrating to something else, but so far, as much as my co-founder hates me for having to learn PHP now and what's with all the dollar signs everywhere, we work with PHP. As for DigitalOcean, it was a very interesting story and a lesson learned. Initially, AirCare was hosted on Amazon service,

I'm not sure what their pricing models are now, but back then they were traffic-based, so the more traffic you got, the more you had to pay and it would scale automatically. This was in the first year or two of AirCare's, let's say, launch locally here. It was only a small app here in Macedonia. Nobody else outside of the country even heard about it. It wasn't even tr1anslated in a different language. The local politicians heard about it and they were not happy that somebody was talking about air quality that was there.

One day, I try to open the app and I see it's not working, so I get confused, why is it not loading? I pop up the server, I look at the logs, and I see that the server disc has filled up, which is weird because that should not happen. It was a big disc. I started looking into it and I realized that we were in day three of a DDoS attack. The app was under attack for three days straight, but because Amazon servers can take the attack that a country of two million can forge up, so it's nothing for them, they were quite fine, but the logs actually in the server filled up and clogged up the server so the server can no longer process.

I freaked out, okay, wait, what? The app is now under attack. We got to do something about this. Especially as a novice developer at that point, I really didn't know much about security and how to protect an app. I mean, to that point, I was still making GET requests with a password in them. It was a thing that I needed to learn very, very quickly. I sat down, I shut off the server, and I had to figure out ways of putting protection like CloudFlare on it to migrated to a host that will not charge me for traffic, to put a lot of caching on top so that not to expose my endpoint so they can't directly attack them.

It took three days for me, basically locked in my room and only eating and coding to fix this because this was my baby. The app is my baby, and someone's attacking my baby so now I have to figure out a way to protect it. After three days, I leave my room. We've migrated to DigitalOcean at this point, but I have, I think it was $600 bill in my hand from Amazon. Amazon saying, "Hey, look, that's the traffic you got, you got to pay." As a student in Macedonia in Eastern Europe, $600 is a lot of money.

Luckily, I had something saved up from some jobs that I've been doing, so I could pay the bill, but that would basically leave me bankrupt. I put a pledge on Facebook and said, "Hey guys, listen, this is what happened. Here's the server logs for those who want to look into it and probably maybe see where the attack is coming from," but then who do you call? You call the police and say, "Hey, listen, stop attacking me." We put it out there and I asked people, "Hey, could you donate a couple of bucks? Here's my PayPal. If it works, great.

If not, at least for me to cover the bill." I came back six hours later, I think I was outside trying to catch a breath, and I came back and there were over $1,500 in donations, which might not sound a lot, but for here, again, for such a small country with such a low economy level, it was crazy. At that point, I kind of learned that, hey, listen, people saw AirCare as their app, it was no longer my app, it was the app of the community, and they did not like if someone was messing with it. From a hard lesson learned, this is when we started to migrate to to DigitalOcean, just because their pricing was much more affordable. Just because even if we got hit by DDoS attack, we wouldn't be bankrupt, even though, of course, they have models of where you can scale the survey if you so wish. Now years after, with more experience, I still am happy to be on DigitalOcean just because their customer support is, I don't know, in my opinion, unbeatable. They're very, very good at helping you out, whatever type of problem you have.

Jonathan: How many servers or VMs are you using, is it a single server or do you have some distributed workload there?

Gorjan: We're using a whole one server.

Jonathan: A whole one server?

Gorjan: Whole One Server to do all of this. We're slowly starting to distribute all of this. The second guy that I have on board and my co-founder who is a techie he came here and he wrote a big list of knows that I've put down in the architecture is like, okay, microservices, we get rid. Each server has their own thing, so it's to not clog everything else. We're in the process of distributing this to multiple servers now because there's a lot of computation that goes into this.

There's one thing to grab the data. There's a whole different process that serves the data. There's a third process that generates, I don't know, heat maps that you can see in the app, which again is quite a cumbersome process because it involves huge amounts of data that are being processed. We're trying to distribute a lot of these into smaller servers that even if they go down, it won't drag everything else with it. Finally, he convinced me to get the databases at least off of that one machine so we have managed databases that we said, okay, you know what DigitalOcean, you handle our databases.

I don't want to sit down and have to learn how to manage a database server at this point. We're using a managed database MongoDB database there. We used to have MySQL Database that we kept all of our data and measurements on, but we realized that with the huge amount of data that we have, it simply would not scale well and it simply wasn't as effective as MongoDB as in querying all of these huge time bound and location bound data. This is why we chose to migrate to a different database.

Jonathan: What does your software development cycle look like? Let's say you have a new idea for a great new feature or maybe it's a bug fix. Walk me through the steps it takes from the idea and tell it's in front of customers.

Gorjan: Got it. Let's take a new feature, for example. With the new feature, it would come down through one of our weekly meetings, probably where someone would pitch this idea and say, hey, look, this seems a great thing that we want to develop. We would sit down and we would do some quick research on it. Let's see, I don't know, has anybody else done something like this? What's the competition looking at this? Is there any point in doing this? If it's a bigger feature that requires more coding, we actually start doing user interviews.

We're working on a really cool health feature right now, but because it is very big in technical terms, we can't simply put out a prototype and see if people like it. We have to do a lot of user research, user interviews, trying to get this down. After we have that, we will sit down and we start figuring out, okay, how do we call this? What parts of the system are being touched? He probably split this up into me and my other co-founder. I take one area and tackle, he takes another area and tackles it, and then we would come together and connect this.

Luckily, when I was starting developing on AirCare back in the day, I was scared of Git, and I was scared of any version control system so I would have a USB, which I would plug into different places. We don't do the city anymore, which is great. After we work on our separate branches and start merging all of these things and checking out each other's code, we would do very short code reviews. Me being more experienced in the current code base would review how my co-founder's code connects to all the services, and then he would review my code from a mode code smells and code practices type of way.

Once we both agree that everything is good, this will get merged into, if it's a server feature, would get merged into the server database and then deployed. We have automated scripts that would monitor if something goes wrong. I'm working now on a script that, of course, will also revert in order to stop. If we push something out and we didn't catch it on time that it's a problem, just automatically revert it to a previous date so we don't impact too many users. That's kind of a small pipeline of everything.

We have monitoring tools that are set up that connect to Slack, that connect to email, that will start freaking out if something has gone wrong. This is talking from a server perspective. From a client perspective on the mobile apps, it's a more cumbersome process because then once you have the code, let's say, and get agreed on, we have to test on multitude of devices. We have a bunch of devices that we test on from iPhones to Androids, to Huawei phones that don't have Google services on them that you must know that it doesn't crash.

Different versions of operating systems. It helps that, for example, you can test on emulators for Android, but for iOS, there are no emulators. There are simulators on your Mac that just simulate an iPhone environment, but do not emulate the actual thing. You'll see how it'll look like, but you're not 100% sure that this is how it'll act. You need always a physical device to test. From that point on, it gets pushed out to all of these different App Stores that AirCare is deployed on. Depending on the App Store, it takes a while for it to-- from anywhere from an hour to a week to get approved if there's any manual approval process and to get launched. That's how it looks like. It's not as sophisticated as bigger companies.

I used to work for Booking.com in Amsterdam for four years as a software engineer. There, we had much bigger release cycle process and deployment and it was very hectic. As a startup, we tend to minimize the overhead that we have so we can go faster and employ the idea of fail fast. If we're going to do it, we might fail, but we'll fail fast, figure it out, fix it, and continue with it, rather than put, I don't know, 20 different types of systems that'll slow our develop into a crawl. Because when you're a startup and you're fighting the competition which have raised millions of dollars, you really need to be fast. If you start very slow, they will eek you up like a cupcake.

Jonathan: On the server side, what kind of automation, if any, do you have in place for deploying in an update? Is it an FTP process that you manually trigger to update the server, or do you have like a Blue/Green Deployment Process, or how does all of that work?

Gorjan: Currently, it's actually an FTP process. It's very basic. Now we go to Git, and then with hooks, it tries to deploy directly on our server. At least, we don't get very desynchronized between what we have on our local machines, what there's on Git, and then what's on the actual server and that's running. We tend to make sure that whatever's on Git is production worthy, at least on the main branch is production worthy, and then that gets automatically deployed to the server.

Jonathan: Is the server essentially running a Get checkout?

Gorjan: Yes.

Jonathan: Okay. I've heard Dave Farley, I think, say recently on an interview that he remembers the good old days when continuous delivery was FTPing one file to your PHP server or your Pearl server, whatever. People like to crap on these languages, but they do give you a certain amount of flexibility for this sort of thing.

Gorjan: Of course, this is great. Even if there's something that's really going on bad on the server, we can FTP it. Get in there, you FTP, you deploy, it's there. This is why I'm saying it allows us to move fast. Yes, it has its negative sides, but there's something about, or why they call it the good old days. There's good in there as well, and that's why we stick to it.

Jonathan: I like what you said, "There's good in there." Not everything about those days were good, but there was definitely good in there.

Gorjan: Exactly.

Jonathan: What's your strategy? You've talked about some manual testing, especially with all these different mobile devices and so on. What sort of automation do you use for testing, if any? Do you have unit tests on the server or on the front end? What does that look like?

Gorjan: From a server perspective, yes, we have some basic unit tests that I wouldn't call them unit. They have more integration tests just to make sure that everything else is connecting to everything else. We have a mock environment where we initially deploy all of these things and see how it works and see if any of the scripts start to fail or implode in our face. We have a mock environment at the app, which of course connects to the milk environment of the server.

We have a whole separate testing, staging area, however you want to call it, Vanguard area, where we play along and make sure first initially that it works there. After that, once we we're satisfied with the manual testing and once we're satisfied that the scripts that automatically run do not show any signs of failure, the-- I'm talking about the scripts so these are not scripts that are written specifically to test. These are the actual scripts that would run on the actual production server. They just run on our testing server and we monitor, see if anything explodes, if anything goes down.

We have a lot of logging that's included there. Any little thing that happen, it'll start triggering warnings. Once we have that, then we do the manual testing on the devices. If all of that is well, then we do a stage rollout. Every time we try to do a stage rollout, especially on Android, because it gets released faster and we can faster fix something than on Apple-- Apple takes a longer time because they have a manual review process. We do a stage rollout, 10%, 50%, everything good 100%. That's not an extensive testing framework, and it has gotten us sometimes in trouble where we've missed something something big. I remember at one point it was a very ridiculous thing. It was not a syntax error. It was not any sort of scriptor. It was a logical error in code. Those logical errors, yes, you can catch with good testing. We didn't have anything back then. What happened was I deployed a version of the app that did not have ads on it for some reason. I think that the if statement was wrong and I didn't see it, it was great.

Then suddenly, I see day after day, the ad revenue very firmly start to drop, and I'm asking what's going on? Why is this happening until I realized there's no ads on the app. I was halfway across the city, I had to rush back in through a quick re-deployment. Adding more tests, and that's something that we're trying to do now. The problem with these things is you're limited on the amount of people you have on the project. Until recently, it was me.

If you have one guy trying to do back-end, front-end app development, server management, and so on, there's only 24 hours in the day, so you start to cut corners. You say, okay, you know what? I'll test manually and then we'll do it live basically. We'll let users test live. Now that there's more and more people coming on board, we're able to put on these quite well-defined software practices and make sure that everybody's at least some automated tests that we have that are running in the background.

Jonathan: You've mentioned logging and monitoring a couple times. I'm curious what you have in place there. How will if the ads aren't working next time or if the server's completely offline or whatever? When something goes wrong, how do about it?

Gorjan: We tend to use a multitude of tools for that, from very basic stuff like Uptime Robot, which just pings the side and sees if it goes down or not, to tools from DigitalOcean itself to monitor the stats of the server, see if we're hogging memory, see if CPU's going crazy, see if there's latency. For example, we recently discovered one of these issues that we had a lot of latency during a certain period of the hour, only because we instated these checks that came from DigitalOcean on that automatically look at these things.

Then finally, we have Loggly which is basically a service that just looks at the logs of the server pulls out specific ones which you filter for. Every time that we have logging from our own scripts, we tag them with specific tags that this service would look for, take it in there, and if something starts to appear, would then ping us on our Slack server. We try to use Slack as our go-to thing, hoping that Slack doesn't go down. Because if that goes down, we have to figure out different ways of monitoring.

That's the problem when you start to rely on one service to the other service to monitor stuff. Sometimes it comes down to just going onto the server and looking at the logs yourself and just calming through all the raw logs and seeing, is there something out of place here? Is there something that's telling me that I haven't caught with any of the other services that might be an issue? We try to automate all of this because simply you don't have time to sit down and look through server logs all day.

It's a combination of a couple of these services that would monitor all of these. Of course, Google has some nice things as well. When you deploy apps on the Play Store in beta testing, beta or alpha testing wherever you put these things, it would automatically test the apps for you. It does some very basic testing but it'll tell you-- it'll just randomly click on stuff and try to go to different screens. It'll try to do a lot of these things and it'll tell you if the app crashes and you get videos, you get screenshots, you get all of these things. Even if our logging misses something, then through the deployment process, it might get caught.

Jonathan: What other challenges have you been surprised by, if any, in the last few years as you've been working on this?

Gorjan: Some of these things, for example, we were implementing a service for-- we did a whole redesign of the app earlier this year. We decided, hey, listen, the concept of the app working per country, that's how it used to work. You select a country in the app and then it'll get the data for that country. If you want to check somewhere in a different country, you have to select a different country and it'll reload the whole app. There were a bunch of other service that were connected. We realized this doesn't make any more sense.

We're going global. We need to make sure that you can look up any place from the app without it having to reload and reload a bunch of different files that are cached and so on. One of the processes in this whole redesign was that we needed to be able to look up any place in the world. We couldn't rely on the names of the stations that we had set manually before. You had to have a geolocation service. We looked online, we found, I think they were called Location IQ. It was one of these services. We said, hey, they have a great service.

We can use this, we can put it there. We connected, we launched it. What we didn't do at that point was we didn't care to calculate the amount of requests that this API was going to get hit by, especially during surge times, if there's a wildfire going on in California and suddenly everybody's on the app looking at how bad the air pollution is. At one point, I opened the app, and again, it's struggling to load and I was very confused why. These guys reach out to us and they say, hey, listen we see that you're exceeding the quotas probably two times than you should be.

There's something apparently that you've miscalculated here. We'll give you a little bit of a wiggle room, we'll give you some free credit, but you got to get this thing sorted out. That was one of these struggles where we realized, oh, wait, if you scale very fast, a lot of these services that you might be using that have quotas will breach the quotas. Unless it is set up that they will automatically either upgrade you to a different tier or will give you some wiggle room with free credit so you can figure out what you want to do, they might bring down your service that you're working on.

From that point on, we said, let's see what are these dependencies that we have? What are their rate limits? Then see even if they go down, can we make sure that it does not bring down the whole app? It does not bring down the whole experience. It might say, hey, I don't know what your current location name is, so we'll just call it current location, but better that than it not loading in the first place. I think it was, again, a valuable lesson for us to learn.

Listen, if there's a lot of these, you will have surges, for example with us, we know we'll have surge moments, and we need to make sure that we can survive these surges. Now we're looking into ideas of how to stress test basically the server, how to stress test the databases, how to stress test everything to make sure that if we launch in a new market and there's a lot of people that suddenly get on board, we don't want to-- it's a nice problem to have that you crash from a lot of users but, again, let's not have the problem in the first place.

Jonathan: That's a great segue into what I wanted to ask next which is, where would you like to see improvement? Where would you like to focus effort on improving the technology?

Gorjan: I think what we're starting to do right now is overhauling the back-end. The back-end is something that was initially written eight years ago by an inexperienced programmer called myself. Then as I got better and as I started learning more and more about software development practices and the ways that these things are handled, getting experience from working at Booking.com, which had a huge code base, I started implementing improvements. Improvements from time to time, you get more and more implementation.

Sometimes you would rewrite whole services, sometimes it would be just patchwork. At some point, you got to say, listen, we're done with feature development. Even though that's something that I love. I don't know if maybe at night I'm a product manager and I love features, but at some point, I got to stay stopped and my co-founder as well who's very technical says, stop. We have to recode the back-end because we've learned a lot now of how it is to better structure all of these flows.

We've learned how to do a lot of automation. We have learned how to do a lot of basically all of these cool things and cool features we've learned throughout the years, which would require us to recode the back-end. The good thing is, recoding of the back-end does not take a lot, because every year, I have these cycles where my OCD kicks in and I say, listen, I need to clear all the clutter out.

I start cleaning out all of the back-end. There's some dead code here and there, files that are not being used, and we're still keeping them the back-end quite slim. The moment that it comes to recode this and make sure that we're using maybe a better framework because right now even in PHP, we use our own little framework to use a better framework, to automate testing. Then I think both my internal developer and my co-founder would be very happy with the outcome that we will have.

Jonathan: You were talking about all these sensors that you're querying and that some people can have their own sensor. What's involved in setting up a sensor? If I want to put one in my backyard, what do I do, and how do I get it integrated with your app?

Gorjan: Luckily, there's a guide within the app.

Jonathan: Nice.

Gorjan: In the app, there's a big button that says add sensor which will tell you. It comes down to where you are and what's available to you. In the US, for example, there's a company called Purple Air and they sell these kits. They're pre-made sensors. You just buy them. You don't have to do anything yourself. You buy it, you hook it up to your WIFI, it starts working from there on. It's very easy to use. From that point, it gets connected to their servers and from their servers, automatically, we pull it in. We have scripts that automatically add new sensors all the time. You basically don't have to do anything. You plug it in 24 hours later, it's on the app, it's there. In other parts of the world, for example, in Europe, there is a network of volunteer sensors called Sensor Community. Now, Sensor Community works with little-built sensors that you can build yourself. You order the parts from either a local manufacturer or from AliExpress, for example, from China. They would come, you would put them together.

It requires a bit more work because you plug in some cables and then you flash some firmware, so it's a bit more techy. We are actually doing workshops here with students in high schools and teaching them how to do this, so it's definitely not very hard. You don't have to weld anything. You put this all in some PVC tubing, you strap them onto your backyard, and then you have a station that starts to measure the air quality there. Now, there is a certain problem with data quality that is involved, especially when you have people putting up their own sensors.

Because nothing stops me from taking that sensor and putting it in my bedroom, putting it in my basement, putting it on the 45th floor of a building. The sensor can't know. It does not know where it is. It's inside. It's outside. We have guides which tell people, "Hey, it has to be outside. It can't be in a little air value we call it, so you can't put it inside of your balcony. It has to be outside. It has to have some air. No higher than the second floor, and so on." We also have algorithms to keep track of this.

If our system detects that one sensor in a particular area is deviating significantly from all of these other sensors, it will start flagging that sensor and saying, "Hey, listen, this seems like an unstable data source. You might want to not take this into consideration." If this keeps being flagged for a longer period of time, then our system automatically turns it off and says, "Okay, we're not using this data source anymore, it seems unreliable."

From time to time, we would ping it again, check if it's okay. If it's okay, it'll turn back on. Usually, it tries to keep it off, because, from 35,000 sensors, it's very hard to-- you can't go outside and check all 35,000 sensors in their place around the world. You have to rely on algorithms that'll make sure that even if people misplace or mishandle them, the users and the data won't be affected.

Jonathan: All right. Is there anything that I have missed or that I should have asked you about that you wanted me to ask you about that I didn't?

Gorjan: I think you did a really good job of covering it. One of my points of creating AirCare apart from trying to save the environment because according to data that we found, seven million people every year lose their lives prematurely due to air pollution. That is 3.5 times my country population. It is huge number. It is a big number here in the Western Balkans. Even in Eastern Europe in general, it's a big problem. One of the ideas was to raise awareness to get people to talk about it.

The second idea, which I wanted to do is to show that even from this tiny little country, which most people struggle to find on a map-- we're very small, you can't even put a pin in there. We're very small, but we're on top of Greece, for those who are wondering. If you know where Greece is, we're right on top. We're the next-door neighbor. Even from this small country, you can make something that can help millions of people around the world. This was our goal to motivate people here, to motivate young people to try and build their own startups, to work with technology, and it's working.

A lot of people here are getting interested in technology. A lot of companies are coming. A lot of devs are being born and being taught here. In a way, it makes me happy that apart from trying to help people know what they breathe in and saving the air quality, we're also promoting the idea that you don't have to be Silicon Valley-based company in order to succeed around the world. You can come from this tiny city and this tiny country in the middle of nowhere and still make an impact on a global scale.

Jonathan: Is your company struggling with software delivery? Would it be helpful to bounce some ideas around with somebody who's been there and done that before? You can borrow my brain for a one-on-one consultation call, go to jhall.io/call for all the details. Will you tell our listeners where they can go to find the app, or learn more about the company?

Gorjan: Sure. There's two sources. One, you can go on our website, which is getaircare.com. If you type getaircare.com, you can read more about our story, where we started from. We have a TED Talk there, which you can look at if you're more interested. I gave a Ted Talk presentation on this whole concept and of course, you can download the app. If you're more keen on checking it on your phones, it's free. It's in the App store, it's in the Play Store. You can go there. It's called AirCare. If you type Aircare in any of the app stores that are available out there, you will find it. You can download it for free. You can play around and, hopefully, it will help you know what you breathe and it will make sure that you're more aware about what is in the air around you.

Jonathan: If anyone's interested, if they have questions that I didn't ask and they want to contact you directly, is that permitted, and how can they find you?

Gorjan: Yes. You can totally do this. You can ping us through any of the emails you find on our website on getaircare.com. All of them go straight to my inbox anyway, so no matter which email you will choose, I will read it and I would be glad to help out. I believe in helping and giving back to the community. That's how we've come so far because of the community aspect and this is why we love giving back to the community.

Jonathan: Wonderful. Well, thank you so much for coming on, Gorjan. I've really enjoyed this conversation. I've learned a lot. I hope you've enjoyed it.

Gorjan: It was wonderful. Thank you so much.

Jonathan: To all of our listeners, we'll see you next time.

[music]

[00:55:58] [END OF AUDIO]