Integration Testing Difficulties with Flutter

Want to know the real struggle of automated testing? Try to cobble together automated integration testing for a Flutter app! It's like trying to construct a puzzle where all the pieces are from different sets and some of them are missing.

Integration testing is an essential step in the development of mobile applications, but it can be especially difficult when using Flutter. Flutter is for multiplatform development and as such it's an abstraction layer on top of many others, including Android, iOS, and web. In their rush to get products to market, platform makers seem to develop their testing story as an afterthought. Flutter is no different and in fact, it's worse, making its testing story an after afterthought!

My trials and tribulations of integration testing with Flutter:

Flutter's recommended integration test mechanism fails to run on Google Firebase Test Lab and Google Firebase Support does not have additional help they can provide apart from filing an internal ticket.

I tried to get Flutter's integration tests to run on an emulator on GitLab Continuous Integration (CI), but I could not find a Docker image that had all of the dependencies. There were articles about the dependencies needed, but they were daunting and required ridiculous tricks such as: "After you run ./android-wait-for-emulator, consider waiting for another 2-3 minutes before starting the tests". Even if I got this working, I would probably spend an infeasible amount of time to maintain it.

Since I am familiar with Android's Espresso test framework, I tried that but it has no visibility into the Flutter view (FlutterSurfaceView) so it can't simulate any actions nor validate anything.  In the screenshot below, the FlutterSurfaceView is a blackhole to Espresso:

However, Flutter does have a bridge, the Flutter Espresso plugin.

Unfortunately, the Flutter Espresso plugin conflicts with Flutter Firebase SDKs. I need Firebase SDKs for authentication, data storage, messaging, etc.

The Flutter Espresso plugin was also hard to learn because I'd have to import a bunch of their equivalents of familiar Espresso functions. Even when I got it to work on Firebase Test Lab without Flutter Firebase SDKs, it was hard just to figure out how to get it to validate a piece of text exists on the screen.

Having invested about a week into implementing a reliable integration test framework for Flutter, I've decided to fall back on manual QA testing and unit testing. That's a shame because on other projects I've gotten a lot more benefit from automated integration testing than other forms of testing. It makes it easier to maintain older projects and gives me the confidence to refactor large swaths of code and even to rewrite entire projects from scratch.

Others agree that automated integration testing is very valuable:

"One thing that it doesn't show though is that as you move up the pyramid, the confidence quotient of each form of testing increases. You get more bang for your buck. So while E2E tests may be slower and more expensive than unit tests, they bring you much more confidence that your application is working as intended." 

- Kent C. Dodds

"Write tests. Not too many. Mostly integration."

- Guillermo Rauch

I wish there were a cloud service that could test on real devices by a robot that can:

1. Tap or click like a human

2. See, hear, read braille, etc. like a human

3. Think like a human...but not too much