Android: Pros and Cons of Loaders

Loaders are weird. The idea seems simple, but just doesn’t seem to stick with you. Perfectly capable Android developers show no shame in saying they don’t understand Loaders. So I took some time to look into them (again) and write it down for future reference.

Simplest implementation is AsyncTaskLoader. Example in documentation is MSDN-style, meaning it’s comprehensive, but you won’t understand anything. So here’s the most basic implementation.

protected void onCreate(Bundle savedInstanceState) {
    getSupportLoaderManager().initLoader(0, null, callback);
}

private LoaderManager.LoaderCallbacks callback = 
        new LoaderManager.LoaderCallbacks() {
    @Override
    public Loader onCreateLoader(int id, Bundle args) {
        return new MyLoader(SecondActivity.this);
    }

    @Override
    public void onLoadFinished(Loader loader, Integer data) { }

    @Override
    public void onLoaderReset(Loader loader) { }
};

private static class MyLoader extends AsyncTaskLoader {

    public MyLoader(Context context) {
        super(context);
    }

    @Override
    protected void onStartLoading() {
        forceLoad();
    }

    @Override
    public Integer loadInBackground() {
        return 10;
    }
}

In this example:

1. Activity calls its LoaderManager to register loader with id=0 and given callback
2. Callback then creates the loader when necessary
3. Loader does loadInBackground(), supposedly doing an expensive operation

One confusion I have every time, is it’s necessary to call forceLoad() manually, or the Loader won’t do anything. This is apparently because there could be some tricky scenario where you already have data to return even before the loader ran for the first time. In most simple case, you just need to know that onStartLoading() is called at the right moment to start the background thing.

Here’s how it works.

D/test: Activity.onCreate()
I/test: initLoader()
I/test: onCreateLoader
I/test: MyLoader()
I/test: MyLoader.onStartLoading()
I/test: MyLoader.onForceLoad()
I/test: MyLoader.onCancelLoading()
I/test: MyLoader.loadInBackground()
I/test: onLoadFinished 10
(rotate)
D/test: Activity.onDestroy()
D/test: Activity.onCreate()
I/test: initLoader()
I/test: onLoadFinished 10
(return)
I/test: MyLoader.onStopLoading()
I/test: onLoaderReset
I/test: MyLoader.onReset()
D/test: Activity.onDestroy()

So this is very intuitive: Loader survived activity rotation, delivered same result to the recreated activity, and then reset itself when activity was about to destroy.

Now let’s make the background operation longer.

public Integer loadInBackground() {
    try { Thread.sleep(5000); } catch (Exception ex) {}
    return 10;
}

Results:

D/test: Activity.onCreate()
I/test: onCreateLoader
I/test: MyLoader.loadInBackground() started
I/test: MyLoader.loadInBackground() finished, isReset=false
I/test: onLoadFinished 10
(return)
I/test: onLoaderReset
D/test: Activity.onDestroy()

When Loader has time to load everything, it just delivered result normally and then died with activity.

D/test: Activity.onCreate()
I/test: onCreateLoader
I/test: MyLoader.loadInBackground() started
(rotate)
D/test: Activity.onDestroy()
D/test: Activity.onCreate()
I/test: MyLoader.loadInBackground() finished, isReset=false
I/test: onLoadFinished 10

Activity was recreated while Loader was working. It just continued and then delivered the result to the new activity.

D/test: Activity.onCreate()
I/test: onCreateLoader
I/test: MyLoader.loadInBackground() started
(return)
D/test: Activity.onDestroy()
I/test: MyLoader.loadInBackground() finished, isReset=true

Activity was closed while Loader was working. It didn’t stop magically (because we didn’t check any flag anywhere), but on completion it didn’t deliver any events and apparenly just died silently.

One more interesting test is to deliver “content changed” event and see how Loader does.

public MyLoader(Context context) {
    super(context);
    new Handler().postDelayed(new Runnable() {
        @Override
        public void run() {
            MyLoader.this.onContentChanged();
        }
    }, 7000);
}

public Integer loadInBackground() {
    try { Thread.sleep(5000); } catch (Exception ex) {}
    return new Random().nextInt();
}

Results:

D/test: Activity.onCreate()
I/test: onCreateLoader
I/test: MyLoader.loadInBackground() started
I/test: MyLoader.loadInBackground() finished, isReset=false
I/test: onLoadFinished 1303887722
D/test: Handler.run()
I/test: MyLoader.loadInBackground() started
I/test: MyLoader.loadInBackground() finished, isReset=false
I/test: onLoadFinished 1662614671

So without anything going on in the Activity, Loader received “onContentChanged”, restarted its background thread, and delivered new result. Now let’s deliver this event while first background thread is still running.

D/test: Activity.onCreate()
I/test: onCreateLoader
I/test: MyLoader.loadInBackground() started
D/test: Handler.run() isReset=false isAbandoned=false
I/test: MyLoader.loadInBackground() finished, isReset=false
I/test: MyLoader.loadInBackground() started
I/test: MyLoader.loadInBackground() finished, isReset=false
I/test: onLoadFinished 474091183

Apparently Loader understands that content has changed while it was still running, and instead of delivering first result, it just re-runs the thread and just delivers the second one.

Also, you may have noticed that I’ve changed “10” to random number. This is because when Loader sees “10” in the result every time, it doesn’t deliver it. Apparently it thinks that Activity doesn’t need the result unless it’s changed.

One more important thing I found in testing, is Loader still contains a strong reference to Activity. So Activity can’t be dropped while background thread is running.

Conclusion:

You might remember how at first recreating activity on rotation was presented as The Shit. And then everyone started hatin on it, because it made it unreasonably difficult to manage data. So AsyncTaskLoader, in its simplest form, is essentially a hack over that. It allows us to manage background requests as if rotated activity never recreated and was the same object. In a perfect world, it would just be this way to begin with, and then we wouldn’t need it.

Suppose you have these conditions:

•  you just need to load data once (like from DB)
• reacting to someone else changing the data is not necessary
• your activity does not recreate on rotation
• or you have a fragment with retainInstance

Here, you don’t need anything other than a single thread started onCreate. Loader won’t give you anything on top of a simple AsyncTask. It just provides some small gimmicks, which are pretty easy to recreate with Activity.isDestroyed(). Remember that Loaders are hard, people don’t understand them, and you want to keep things simple.

On the other hand, in following situations you might want Loader:

• you have classic rotating/reloading activity
• background operation is long, data may change in the process, and you want to react correctly on change events

That’s where you actually have some not so straightforward cases of managing threads and results, and you don’t want spend time and rewrite what Loader already provides.

CursorLoader

I can’t just close this topic without due hating on CursorLoader. I am very much against the very concept. It seems like a natural evolution of a regular Loader, and almost the first intended use case for it. However, it does stuff which is very, very harmful for the code.

What CursorLoader does, is it gives UI classes direct access to a DB. It kinda tries to obfuscate that by running data through a ContentProvider first, but in the end it’s still the same DB access methods. End results of that:

• Activities now know how your DB looks. You can’t ever change anything about your DB structure, because it would require changes in 50 Activities and related helper classes.
• You have an intermediate ContentProvider class. It does essentially nothing but mapping CursorLoader logic to DB. IF YOU LUCKY. I’ve had cases for example where ContentProvider didn’t implement “delete” (because it was never needed), and I spent half a day debugging DB and trying to understand why a perfectly normal operation doesn’t work. And it’s a relatively simple example, it’s possible to break ContentProvider in a way you wouldn’t believe.
• Despite of what you might expect, using CursorLoader doesn’t mean you magically get memory benefits. What Cursor does, is it first loads the entire data set into memory, and then just gives you a way to conveniently iterate over it.

At best, CursorLoader is a low-level optimization for low-powered devices. Initial idea was that you just read numbers straight from a DB row and put them directly into UI. Forget all that MVP crap, we’re old-school. This nice feature comes at a cost of a huge added complexity, and basically making sure only advanced developers will be able to work with your code. Because have you ever tried to work with a huge chunk of code which doesn’t even have any business objects and just passes god damn Cursors everywhere?

Instead, I usually create a “data manager” class, which completely isolates any access to DB. If you want any data from DB, you go through that class. It loads the same dataset CursorLoader would, but then converts it all into business objects and closes the cursor (releasing the underlying data). Am I rewriting ContentProvider? Yes. Yes I am. But let’s compare them.

ContentProvider:

• Low level data model, presents data as tables of primitive values.
• Users use Cursors to navigate the data. If they’re good, they then convert rows to business objects themselves. If they’re not, they either use primitive values straight from a Cursor, or worse pass Cursor somewhere else, completely ruining its lifecycle.
• Returns just the rows user requested.
• Built-in functions for changing and observing data, documented in SDK, require everyone to use this ContentProvider to access this DB.

Custom data manager:

• Object level data model, presents data as collections of business objects.
• Users receive List<BusinessObject>, which is usually end result of what they wanted.
• In simplest implementation, returns the entire dataset for user to navigate.
• Custom functions for changing and observing data, discoverable through code, require everyone to use this manager to access this DB.

So here I lose some on performance, but gain in presenting data to the outside users in a way which people will actually be able to understand. This approach is unfortunately completely incompatible with CursorLoaders, which makes me sad when I have to have discussions with people who like them.

Android: Image Cache

This is more of an announcement. I’ve uploaded a new opensource project, ImageCache. As the name suggests, it’s a component for Android that handles downloading and displaying images from the Internet. Basic call is like this:

CacheQueue.displayImage(imageView, "http://i.imgur.com/u0NjkvC.jpg", 600);

With this call, the component will download file from given URL, save it to disk (I kinda like saying disk, it’s not a disk though), open with dimensions around 600px on the longest side, and assign to the imageView. If the imageView somehow became irrelevant in the process (say the activity reloaded), then cache wouldn’t keep it and will just store the image. Naturally, subsequential calls for that same url will either pull the image from memory, or from the disk cache.

This sounds simple so far, and you might wonder why wouldn’t I just use AsyncTask for that like a normal person. As it turns out, there is more.

1. What if we make lots of calls at the same time? In this case there has to be a strategy as to how many download threads to run. This cache is built on top of a custom LoaderTask component, which sets it to 5 concurrent threads. It is also organized in a way that was meant to reuse LoaderTask for other web operations in the app. You might not find it extremely convenient if you want a separate component, but for me it was a base layer of my app’s web operation.
2. How do we limit the disk footprint? Might surprise you, but it’s really tricky to do right, because you can’t exactly afford to rescan the entire filebase on every sneeze. Most lazy cache implementations I’ve seen don’t even bother. They either just keep downloading files, or delete the entire cache when it becomes too large. This component keeps track of files using DB and deletes older files when it detects an overload.
3. What if we have requested an image limited to 100px (thumbnail), and then immediately requested that same image in 2048px (fullsize) from the same url? This isn’t very hard to implement, but it’s one of those things that can be really annoying and easy to overlook. The component works with that.

Initially I developed this code as part of a Reddit client for a customer, but then I had rewritten it for my own purposes. Not sure if it’s very useful as an actual component (caching requirements often differ a lot), but you’re welcome to check it out if you’re building your own.

Android: SwipeRefreshLayout and requestDisallowInterceptTouchEvent

This is a very specific use case I spent a while debugging.

In our project, we have a ViewPager. Each page inside it is a fragment with ListView as a base. ListView also has reloading via SwipeRefreshLayout. And, in ListView’s header, we had a component that displayed photos. User could zoom in on a photo and pan it.

The problem was, panning on the photo also activated refresh in SwipeRefreshLayout and moved pages in ViewPager. Refresh wasn’t an issue, because at one point we just disabled it. But moving in ViewPager was clearly a deal breaker. What’s strange, the photo component correctly called requestDisallowInterceptTouchEvent() and managed to successfully prevent ListView from scrolling. So why not ViewPager?

As it turns out, SwipeRefreshLayout positions itself like this:

...
 android.widget.LinearLayout
 android.widget.ListView
 android.support.v4.widget.SwipeRefreshLayout
 android.widget.RelativeLayout
 android.widget.LinearLayout
 android.support.v4.view.ViewPager
 ...

So it’s between ListView and ViewPager.

Furthermore, touch events in Android work like this. First, all views from bottom to top receive onInterceptTouchEvent(), and then, from top to bottom, they receive onTouchEvent(). Scroll views, such as ViewPager and ListView, intercept touches to activate themselves and do the scrolling. We can call requestDisallowInterceptTouchEvent() from a view on top to prevent that. Theoretically, every view should pass this call to its parent (further to the bottom). But, as it turns out, SwipeRefreshLayout implements this functionality in a very special manner:

@Override
public void requestDisallowInterceptTouchEvent(boolean b) {
    // Nope.
}

So it just straight disregards the usual practice and blocks this call from propagating any lower.

To work around that, I had to iterate over parents and make the request on ViewPager itself.You might want to consider something like that if you have similar layout in your project. Note however that I didn’t explore any interference this could have caused with SwipeRefreshLayout, because in our case it was disabled for this particular scenario.

Java: memory representation of primitive data types

So I was working on a GIF decoder, and it required some very rapid manipulations with binary data. Converting byte to short, bit shifts, that sort of thing. Obviously, that’s easily done in C. You just cast a pointer to something else, cause GPF, and are done for the day. Java is very big brothery about this stuff. It really doesn’t want you to do any C tricks. So if you need to convert byte[] to int[] – would you please iterate it and do a[i] = b[i] for every one of the million elements. There are of course reasons for that, but it really limits options.

Anyway, one of the issues I ran into was a construction like this:

int[] data = new int[256];
byte index = readIndex();
int entry = data[index];

You’re probably going “ha-ha” now, but shut up, I didn’t know that. So, Java, being a great language, only has signed types. Incidentally, byte is also signed. Therefore, 0xFF as byte is not 255, it’s -1. And this code should have been:

int[] data = new int[256];
int index = ((int)readIndex()) & 0xFF;
int entry = data[index];

So why the & 0xFF? It’s an extra operation. Turns out, here’s how type casts really work.

int n
00000000000000000000000100000001 257
11111111111111111111111111111100 -4
11111111111111111111110000000000 -1024

byte b = (byte)n
                        00000001 1
                        11111100 -4
                        00000000 0
                        
int nb = (int)b                
00000000000000000000000000000001 1
11111111111111111111111111111100 -4
00000000000000000000000000000000 0

char c = (char)n                        
                0000000100000001
                1111111111111100
                1111110000000000
                
int nc = (int)c
00000000000000000000000100000001 257
00000000000000001111111111111100 65532
00000000000000001111110000000000 64512

As you can see, when casting bigger type to smaller (int to byte), it just cuts everything that doesn’t fit, so most significant byte becomes the minus sign. When casting byte to int, it stretches the minus bit to all the added digits. So from this numeric perspective, it’s still the same number. But the entire problem is, in the example above, we want to speak binary, not numeric. And from binary perspective, it’s screwed up beyond recognition (binary being unaware of minus signs). So you have to re-cut the initial part (mask on 0xFF) to make it the same. It sort of works as expected with unsigned types, without this mental gymnastics.

Char is an interesting exception, it’s the one unsigned type. So it’s just always padded with zeroes.

This whole system is not without its benefits I guess, but why did they make the byte signed? I mean come on.

Android: displaying animated GIFs

As it is known, Android does not have a good support for animated GIFs. It’s a whole story with GIF compression patent owners being greedy and destroying this format for everyone, including themselves. In any case, we need to support it somehow.

Great list of available options is given by Johannes Borchardt:

1. Movie class. I don’t like this approach. It’s some kind of sketchy undocumented class, which does not always work.
2. GifDecoder + AnimationDrawable. GifDecoder is a custom class which converts gif frames to bitmaps, and bitmaps are then loaded into AnimationDrawable. Problem with it is that it attempts to decode the entire file at once (that’s how AnimationDrawable works anyway). Which is not necessarily fun when you have 5 Mb gif videos.
3. WebView. We used it a lot. WebView has this thing when it usually works, but never ideally. Once you started dealing with it, you’re signing up to handle tons of its stupid quirks, most of which only reproduce on old obscure devices you don’t have. Plus, you can’t use multiple instances in any kind of list views, because it does things to your memory. So your UI options are limited.

All in all, WebView is a simplest approach, which works for most applications. However, I wanted to look into some more possibilities. GifDecoder looked promising, because it was very controllable. It worked fine for smaller animations. Obviously the entire concept was wrong for gif videos, because there’s no way they’ll ever fit anywhere uncompressed, even on disk. So I rewritten this class for streaming and created a GifFileDecoder project here: https://sourceforge.net/projects/giffiledecoder/ The project includes two parts:

1. Decoder. Same as GifDecoder, it converts gif frames to bitmaps. Unlike GifDecoder, it only needs to keep one or two frames in memory, so it can play any kind of video (I’ve successfully tested it on 15 Mb 640×640 file). If you want to pre-cache bitmaps, you can still do it with same efficiency, the logic will just be outside the decoder.
2. Custom ImageView. It’s a regular ImageView, but it runs decoder in a background thread and does setImageBitmap on itself when needed. It is a somewhat crude approach, it has to be controlled manually, like you have to stop the thread when pausing activity. But there’s also tons of benefits. Like, it’s always clear what’s going on with it, you can pause it at any time and use it as a regular ImageView without any fear of it going rogue and consuming every possible resource.

I have tested the project on my devices, and it appears to run about half as fast as WebView. Which is not great, but also not that bad, considering that Java is not a great language for a decoder, and WebView probably uses some kind of optimization. It runs without delays on newer devices, and starts to lag on heavy videos on something older. But then, by “older” I mean my ultra-budget Samsung phone made in 2011 and running Android 2.3. On that phone, most videos were running on 90 to 100% speed, except some really heavy ones.

All in all, another example of how things are usually really simple once you start looking into them instead of using libraries. GIF format is really not that complex and is well-documented, so it’s not that hard to render manually. C++ would be much more efficient for it, but oh well.

PS: after performance optimizations by Aleksandr Shardakov, decoder runs about twice as efficiently and now shows speeds on par with WebView. Native C++ implementation (see comments) would be even more efficient, but it’s also more difficult to use. So I guess this approach is also not bad.

Android: understanding Widgets

“App Widgets” section of Android documentation describes how to add a widget to your app, and does a good job at that. Unfortunately, it does not describe the generic widget concepts so well. And they are very important once you start actually doing some work.

What’s a widget?

This is actually surprisingly hard to get from various documents. Is widget a separate app? Or is it a part of the same app? But then how does the app work on home screen? Why all the hassle with RemoteViews and such?

In fact, it’s very simple.

In a sense, Android’s home screen is just another app. We do not actually run any code “in home screen context”. Instead, we register with the Home app, and give it schematics for our widget (RemoteViews). Then it creates actual Views according to these schematics. So, we do not handle actual Views, another app does.

// Create schematics
RemoteViews rv = new RemoteViews(context.getPackageName(), R.layout.widget);
// Send to Home app
appWidgetManager.updateAppWidget(widgetId, rv);

Normally we also want to handle onClicks and other events. Usually we just register OnClickListener etc, but these only work within the same app. And Views are in another app now. So, Home app probably also registers these, but instead of running any code in the listener, it just sends us an Intent. Because Intents are basically Android’s idea of cross-process communication. Intents are also defined in the schematics.

// Create schematics
RemoteViews rv = new RemoteViews(context.getPackageName(), R.layout.widget);
// Add intent to fire in onClick listener
Intent intent = new Intent(context, MainActivity.class);
PendingIntent pending = PendingIntent.getActivity(context, 0, intent, 
        PendingIntent.FLAG_UPDATE_CURRENT);
rv.setOnClickPendingIntent(R.id.appWidgetContainer, pending);
// Send to Home app
appWidgetManager.updateAppWidget(widgetId, rv);

In other words, widget is a normal component that inflates Views and assigns Listeners to them. But the trick is, this is all done by another app. We just provide instructions.

Widgets and processes

I’ve seen people work from this concept:

Their idea was that widgets are separate entities that can run in a different process, and therefore should communicate with main app via Intents. It’s easy to make this mistake, but this point of view is wrong.

First, what’s an app, and what’s a process. An app is your project, compiled into .apk file. A process is an instance of your app running in the system. In theory, there could be multiple processes of the same app, running in parallel. In practice, Android always runs any specific app in a single process. There are rare exceptions, but you won’t have to deal with them unless you need to.

To summarize:

• An app can either run in a specific process, or not run at all.
• Any widget-related code will run in the same process as the main app.

As discussed above, widget code is not part of the Home app; it is part of your own app, which just provides some stuff for Home. So, it will run in your process. If your main app already runs and Home suddenly wants something from widgets, widget handlers will run in the same process already running the app. If the app isn’t running, handlers will start a new process, and that process will then be reused if you open the app.

This is very important, because this way you can share singletons, preferences, and other data between app and widgets without bothering with Intents and data synchronization between processes.

Lifecycle of a widget

Note on onUpdate (1). Documentation states that inf you have a configuration activity, onUpdate will not be called before the activity produces any result. At the time of this writing, it’s a lie. It is in fact called regardless of configuration activity, which will cause an unconfigured widget update in your code. To avoid that, people store a flag in SharedPreferences and set it once configuration activity is about to be done.

public class Widget extends AppWidgetProvider {
  
    public void onEnabled(Context context) {
        // First instance of this widget was created
        super.onEnabled(context);
    }
  
    public void onUpdate(Context context, AppWidgetManager appWidgetManager, 
            int[] appWidgetIds) {
        // System wants update on widget(s), on startup, or after creating a new one
        if (!prefs.getBoolean("configured-" + widgetId, false))
            return; // not configured yet
        for (int widgetId : appWidgetIds) 
            update(context, appWidgetManager, widgetId);
        super.onUpdate(context, appWidgetManager, appWidgetIds);
    }
  
    public void update(Context context, AppWidgetManager appWidgetManager, 
            int widgetId) {
        // If you use configuration activity, call this manually after it's done
        SharedPreferences prefs = 
           PreferenceManager.getDefaultSharedPreferences(context);
        if (!prefs.getBoolean("configured-" + widgetId, false))
            return; // not configured yet
        // Send RemoteViews here
    }
  
    public void onDeleted(Context context, int[] appWidgetIds) {
        // User deleted widget(s)
        for (int widgetId : appWidgetIds) {
        }
        super.onDeleted(context, appWidgetIds);
    }
  
    public void onDisabled(Context context) {
        // Last instance of this widget was deleted
        super.onDisabled(context);
    }
}

Naming conventions

This is the part which doesn’t seem at all important at first, but can easily become an annoyance later. Let’s say you got some kind of sample widget working, and are now starting to apply it to your project. It’s almost guaranteed that entire code of the sample is in its main package (such as “com.example.mytest.MyWidget”). Since there’s only a couple of classes, your natural move will be to just drop them in the main package of your project as well (such as “com.acme.explosives.MyWidget”), and maybe even leave it as “MyWidget”, because what’s the diff.

Don’t be lazy. Take 5 minutes to think of a good name for your class, and create a dedicated package for it. For example, create it as “com.acme.explosives.widgets.TNTWidget” instead.

Reason for that is, once you deployed your app, your widget class is pretty much set in stone. When customers add your widgets to home page, system will record their class names, and will not give you a chance to change them later. So, you will not be able to either rename the widget class, or move it to another package, unless you’re willing to break widgets for every customer and seriously piss off your managers. You’ve named your widget “TestWidget”? Congratulations, you’re stuck with this name forever. You’ve eventually created 5 more widgets, and they require like 15 support classes? All that stuff will forever clutter your main package, you won’t be able to isolate it. I’ve seen both of these things happen on more than one project.

So, again:

• pick a good name,
• create “widgets” package right away.

Updating widget with service

A widget can be periodically updated on intervals defined by updatePeriodMillis. However, minimum interval is 30 minutes. Usually people are greedy and want instant updates. So, at least every 5 minutes.

This kind of requirement is implemented with a service handled by AlarmManager.

public class Widget extends AppWidgetProvider {

    public void update(Context context, AppWidgetManager appWidgetManager, 
        int widgetId) {
        // Do checks from previous example
        ...
        // If you don't have updatePeriodMillis, this will be called once on 
        // startup or widget creation. So, start alarm on this first call.
        setAlarm(context, widgetId, 5 * 60 * 1000);
    }
  
    public void onDeleted(Context context, int[] appWidgetIds) {
        for (int widgetId : appWidgetIds) {
            // Remove alarm
            setAlarm(context, widgetId, -1);
        }
        super.onDeleted(context, appWidgetIds);
    }
  
    public void setAlarm(Context context, int widgetId, long interval) {
        Intent intent = new Intent(context, WidgetUpdateService.class);
        intent.putExtra(AppWidgetManager.EXTRA_APPWIDGET_ID, widgetId);
        PendingIntent pendingIntent = PendingIntent.getService(context, 
            widgetId, intent, PendingIntent.FLAG_UPDATE_CURRENT);
        
        AlarmManager am = (AlarmManager) context.getSystemService(
            Context.ALARM_SERVICE);
        am.cancel(pendingIntent);
        if (interval >= 0) 
            am.setInexactRepeating(AlarmManager.RTC, System.currentTimeMillis(), 
                interval, pendingIntent);
    }
}

public class WidgetUpdateService extends Service {

    public void onStart(Intent intent, int startid) {
        int widgetId = intent.getExtras().getInt(AppWidgetManager.EXTRA_APPWIDGET_ID);
        // Process new data
        ...
        // Send updated widget schematics to Home app
        if (haveNewData) {
            AppWidgetManager awm = AppWidgetManager.getInstance(this);
            RemoteViews rv = new RemoteViews(context.getPackageName(), 
                R.layout.widget);
            awm.updateAppWidget(widgetId, rv);
        }
        // Stop service
        stopSelf();
    }
}

The service runs in main thread. So, if you need to download something, make sure to start another thread in the service.

Another very important point which many people don’t care about, until it bites them in the ass. Notice the stopSelf() call at the end. You must do this call when the service is done. It can be difficult, because if you have multiple things to download, you have to predict all possible cases of this service finishing all threads. Like, case 1: nothing to download, stop now; case 2: download thread completed, stop now; case 3: there was an error, stop now. See how annoying this is already? That’s why nobody likes doing it, and you won’t find this call in many samples.

What will happen if you don’t include stopSelf() call, is everything will look normal, but the system won’t actually stop your app. It will be present in Settings – Apps – Running, and will indicate that the app runs this service. For all I know (I didn’t actually check), the system runs CPU cycles for the app the entire time, wasting battery. That’s why it’s very important to always make this call. It’s also a good practice to check if your service calls onDestroy() every time after it’s done.

Android: SQLiteDatabase locking and multi-threading

One of the confusing topics on Android is working with SQLite database. It is simple to use, but there’s almost no explanations on how it works in more difficult scenarios, especially when multiple threads are involved.

I’ll review the usage patterns around SQLiteDatabase first, because most problems start with getting the DB object incorrectly.

Single instance of SQLiteOpenHelper

First thing you need to know is that there should be only one instance of SQLiteOpenHelper in your app. See “Android Sqlite Locking” and “Single SQLite connection” by Kevin Galligan (note: Android source code has changed a lot since then, details may not be entirely accurate). Reason for that is, OpenHelper caches an internal instance of SQLiteDatabase, and if you have many OpenHelpers, you’ll also have many poorly synchronized instances of the same database. I like a pattern which is slightly different from Kevin’s, but does the same thing.

public class Database {
    private static final int DATABASE_VERSION = 1;
    private static OpenHelper openHelper = new OpenHelper();

    private static class OpenHelper extends SQLiteOpenHelper {
        private OpenHelper() {
            super(App.getAppContext(), "test.db", null, DATABASE_VERSION);
        }
        @Override
        public void onCreate(SQLiteDatabase db) {
        }
        @Override
        public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
        }
    } 

    public static SQLiteDatabase getDatabase() { // pattern 1
        return openHelper.getWritableDatabase();
    }

    public static int fetchData() { // pattern 2
        SQLiteDatabase db = openHelper.getWritableDatabase(); 
        // run some queries on the db and return result
    }
}

See my other post about App.getAppContext().

I’m showing you two different patterns in this example. If you only want this class as a DB opener, and will handle the queries somewhere else, then you’re good with just getDatabase(). However, you can also make this class the data access object for the database, and run all queries right here. Then, you don’t need getDatabase(), you’ll want to add a bunch of fetchData()s instead.

Writable vs Readable

Now, you might have a question: why does getDatabase() translate to getWritableDatabase()? Wouldn’t it be safer to return getReadableDatabase() when appropriate?

Turns out, inside OpenHelper, that’s the same object. That’s even mentioned in the documentation. Barring some extremely rare cases, you will always receive the same reference for both. So there’s usually not much reason to bother.

Saved instances of SQLiteDatabase

Another question: wouldn’t it be more effective to save SQLiteDatabase to a member variable, instead of opening it from helper every time?

public class Database {
    private static OpenHelper openHelper = new OpenHelper();
    private static SQLiteDatabase db = openHelper.getWritableDatabase();

    public static int fetchData() { 
        // just use this.db, it's already there!
    }
}

As it turns out, there’s no reason to do it. SQLiteOpenHelper already does the same thing – it has an internal instance of the DB, which it can manage better than you could. Plus, saved instance can cause you problems. See this discussion as an example. So, using OpenHelper is just a safer pattern.

Closing the database

Another non-documented point is, whether you should SQLiteDatabase.close(), and when exactly. Popular answers are Activity.onStop(), and Application.onTerminate(). They are of course both wrong, because onTerminate is not a real event, and you are likely to need your DB in more than one activity. Possible problem is mentioned here, and “Concurrent Database Access” offers a solution based on reference counting. Also, there’s a google groups thread, where a trustworthy source says you don’t need that at all.

To answer this question, I looked into Android code (v4.4.4). Turns out, Android already implements the reference counting internally. Basically, whenever you do anything with SQLiteDatabase, it increments a counter, and when you’re done, it decrements it. When all operations are done, DB just closes itself. I saw absolutely no reason to second-guess this system. Just don’t close the DB. You have to close cursors though.

Multi-thread access

Now that we have a properly initialized DB, next question is – is it at all thread-safe? And if it is, how exactly does it work? Is a transaction thread-safe? What about iterating a cursor? Again, popular answer is “you could use synchronized sections and semaphores”, which really translates to “nobody can possibly know”. Documentation is really vague on this subject. Android source makes an impression of being very thread-safe, but also doesn’t seem to have any apparent thread locks, so from a first glance it looks like that’s entirely up to SQLite’s internal file locking.

To understand this topic, I ran some simple tests with setup like this:

new Thread(new Runnable() {
    @Override
    public void run() {
        Database.runTransaction("A");
    }
}).start();

new Thread(new Runnable() {
    @Override
    public void run() {
        Database.runTransaction("B");
    }
}).start();

Results are actually very simple:

– SQLiteDatabase is thread-safe.

– All operations, including insert, update, and read, are mutually exclusive. Meaning, two operations from different threads can not run in parallel. Transaction counts as one operation.

– Read operations are caching data on query(), before cursor.moveToNext(). Therefore, if query() was completed prior to a transaction, you can iterate the cursor parallel to the executing transaction, and that transaction will not affect the received data. It is even possible to entirely drop a table while a cursor is open on it.

Parallel operations

Rules above, however, only apply to the default Android setup. There are situations when you might want a DB to behave differently. Like, if you have some huge writes, and you don’t want to block reads for them. I haven’t explored this subject much, but here’s what you’ll be looking for.

First thing you’ll try is setLockingEnabled(false). It says right there that it doesn’t work anymore, but nobody believes that. Well it doesn’t. Here’s the precise implementation of this method from the Android source: { } . So don’t bother.

Next you might try to beginTransactionNonExclusive(). Thing is, by default, it works exactly like beginTransaction(). I spent some time trying to figure that out, because it kinda looks like it should allow parallel reads, and it doesn’t. My understanding is, this method only works with enableWriteAheadLogging(). And that latter one is actually the way to go, if you want to allow parallel DB operations. Also check out “Handling some SQLite issues“.

Java: Unescaping HTML entities, or Don’t use libraries

Some time ago I needed to write some Android code which had to convert HTML entities to their string representations. In case you don’t know, HTML entities is the way of encoding charaters where you can write “&” as either “&amp;” (by name) or “& #38;” (by ascii code). This is a well-documented and widely used conversion, and I figured it should already be available somewhere. So there’s Apache Commons Lang jar. Normally I don’t trust 3rd-party libraries, but it’s Apache. It just has to be super optimal and rock solid all around.

So I added this jar and started writing other code. Luckily, I was running performance tests on that. On some point I noticed that performance dropped very significantly. I started looking and found that, even though I barely had any entities in my input, Apache code really took a while to run.

The library is open source, so I looked into what it does. It was quite nice code, but also it did a lot of unnecessary stuff. Not the kind of stuff that handles rare cases – rather, the kind that makes it modular, easy to follow, and generally smart-looking. But not extremely effective. So I spent some time and tried to just rewrite the unescape function in a stupid way. Then I ran this test:

String s = "This is a test string & it has an entity";

Log.i("test", "start test 1");
long time1 = System.currentTimeMillis();

for (int i = 0; i < 10000; i++) {
    String s1 = org.apache.commons.lang3.StringEscapeUtils.unescapeHtml3(s);
}

Log.i("test", "start test 2");
long time2 = System.currentTimeMillis();

for (int i = 0; i < 10000; i++) {
    String s1 = StringUtils.unescapeHtml3(s);
}

Log.i("test", "end test 2");
long time3 = System.currentTimeMillis();
Log.i("test", "time 1: " + (time2 - time1) + " time 2: " + (time3 - time2));

Here are the results, they are rather interesting:

"This is a test string & it has an entity"
time 1: 3421 time 2: 80
"This is a test string - it has no entity"
time 1: 3767 time 2: 5

Now, I’m not claiming my code is as good as Apache’s. My version may not handle some extremely rare corner cases. But so far it has been tested with very rapid calls in a large project, with multiple languages, and there has been no issues.

This is a great example of why I distrust 3rd-party libraries in general. It’s fine if they really do something serious. But free utility libraries doing small tasks are usually not worth it. They just make simple things look complicated and important. Take this entity conversion. It’s really nothing but a 2-screen function plus a data map. But what if you didn’t know that? You’d be stuck with a black box, which takes, literally, thousands of times more resources than it should, and you’d be thinking “that’s just the amount this task takes, nothing to be done about it”. Which is, I suspect, exactly the case with lots of people using this very common library.

Android: Using Context statically and in singletons

In Android, you need context for just about everything. If your code is a part of an Activity, that’s easy. But what if you have some non-UI singleton class, which is used from lots of Activities and other non-UI classes?

Android design documents strongly suggest that you just pass a context to it for every use.

public class Worker {
    public Worker getInstance() {
        // return singleton instance
    }
    public void doStuff(Context context) {
        context.getCacheDir();
        // save stuff to cache
    }
}

public class MyActivity extends Activity {
    public void callWorker() {
        Worker.getInstance().doStuff(this);
    }
}

This is fine, as long as your doStuff() is very simple. But what if you want to save and reuse context in Worker, and your doStuff() is absolutely not guaranteed to be called from an Activity? If you really don’t care, you could just do it like this:

public class Worker {
    private static Context context;
    public Worker getInstance() {
        // return singleton instance
    }
    public void init(Context context) {
        // this.context = context; // (1)
        this.context = context.getApplicationContext(); // (2)
    }
    public void doStuff() {
        context.getCacheDir();
        // save stuff to cache
    }
}

public class MyActivity extends Activity {
    public void callWorker() {
        Worker.getInstance().init(this);
        Worker.getInstance().doStuff();
    }
}

Application vs Activity context

Note the difference between (1) and (2). Activity context is different from the application context, but in this case either will work. However, with (1), we’re leaking the Activity, meaning it will hog the memory for nothing, instead of being garbage-collected in time. Plus, what if later you needed to doStuff() from another activity? You’d be running in a different activity’s context, and who knows what could go wrong. With (2), you’re saving the application context instead. There’s only one, and it lives for the lifetime of your process.

Lots of internet posts will have you believe that you should religiously avoid application context and only use activity contexts. These are written by students who have never written any code outside of an activity. Obviously, application context is fine, you just need to know what it can’t do.

Static context

However, the example above still sucks. Who guarantees that init() will ever be called before doStuff()? What if you want to use context in Worker’s constructor?

This kind of task calls for some static way to access application context. Generally, static context is essential to design a good singleton. A lot of system features ask for context, while having no earthly reason to use anything other than the application. DefaultSharedPreferences and getCacheDir() are good examples. You can use them in a context-specific way, but you won’t do that 99.9% of the time.

However, even though application context is totally static and lives for the lifetime of an app, there’s no way to get it with, like, Context.getApplicationContext(). The problem isn’t technical, it’s just for some reason Android designers didn’t want you to do this exact thing. Probably because application context can’t do some things activity context does. So instead of designing it in a way that would clearly communicate these limitations, they just said people shoudn’t use it.

Common way to work around that is subclassing the Application and keeping a static reference to it.

public class App extends Application {
    private static Context appContext;
    @Override
    public void onCreate() {
        super.onCreate();
        appContext = this;
    }
    public static Context getAppContext() {
        return appContext;
    }
}

Again, in every discussion of this method (like this or this), you will find this obligatory quote from Android documentation:

“There is normally no need to subclass Application. In most situations, static singletons can provide the same functionality in a more modular way. If your singleton needs a global context (for example to register broadcast receivers), the function to retrieve it can be given a Context which internally uses Context.getApplicationContext() when first constructing the singleton.”

That’s exactly the kind of BS I mentioned earlier. Probably subclassing Application could be avoided, if we had a decent way to get static application context. You may also get the feeling from this quote that it’s somehow unsafe. It isn’t, it’s just frowned upon.

There is a downside though. Normally you’ll also want to shove lots of static fields in the App class, to keep your useful global variables and whatnot.

public class App extends Application {
    private static Context appContext;
    private static SharedFoobar foobar1; // (1)
    private static SharedFoobar foobar2 = null; // (2)
    // private static LootCache = new LootCache(); // (3)
    @Override
    public void onCreate() {
        super.onCreate();
        appContext = this;
        foobar1 = new SharedFoobar(); // (4)
        foobar2 = new SharedFoobar();
    }
    public static Context getAppContext() {
        return appContext;
    }
}

public class LootCache() {
    private final Context context;
    public LootCache() {
        context = App.getAppContext();
    }
}

public class SharedFoobar() {
    private static Context context = App.getAppContext();
}

Note that in (3), LootCache was initialized statically. Which means it will run its constructor and getAppContext() before App had a chance to initialize appContext field, which will cause NullReferenceException later. This seems obvious, but in fact it’s really easy to miss, especially if this constructor in LootCache was added later by someone else.

If you’re familiar with normal languages, you may also think that foobar1 and/or foobar2 have this problem too. Since SharedFoobar has a static initializer, and is declared in both (1) and (2), wouldn’t it want to run static constructor right there? In case (2), it looks almost unavoidable, right? So, no, it will not do that. In Java, classes are not even loaded on declaration, they are only loaded “when referenced”. In our case, this means that static constructor of SharedFoobar will only run on (4).

Singleton pattern

Now that we have a way to statically access the context, we can write a proper Worker singleton.

public class Worker {
    private static Worker instance; // automatically initialized to null
    public static Worker getInstance() {
        if (instance == null) 
            instance = new Worker();
        return instance;
    }
    public void doStuff() {
        Context context = App.getAppContext();
        context.getCacheDir();
    }
}

See how doStuff() immediately became easier? We don’t have to pass Context to it from millions of places, or worry about is that the right context, or will it be leaked there, or any of that stuff.

If you’re an experienced dude, however, right about now you should say “ha-ha, you dumbass, getInstance() isn’t thread-safe”. That’s true. You could make it thread-safe like this:

public class Worker {
    private static Worker instance;
    public static synchronized Worker getInstance() {
        if (instance == null) 
            instance = new Worker();
        return instance;
    }
}

But as it turns out, in Java, there’s a fantastic alternative to this universal pattern. Since Java classes themselves are lazy-loaded, you can just initialize the instance statically. It will be loaded on first use of the class and is thread-safe “by design”. Read google groups discussion (subj is “Android LifeCycle and Singleton Instances”) and this post on double-checked locking. (* see Update 1 & 2 below)

public class Worker {
    private static Worker instance = new Worker();
    public static Worker getInstance() {
        return instance;
    }
}

The “null static” case

Let’s circle back for a bit. On this point, you may consider this implementation equally good:

public class Worker {
    private static Context context;
    public static void init(Context appContext) { // only call from App!
        this.context = appContext;
    }    
}

public class App extends Application {
    public void onCreate() {
        Worker.init(this);
    }
}

Since Worker will definitely be initialized in App.onCreate() and only there, Worker.context will always reference Application, right? Turns out, no, it can unexpectedly reset to null. This is a rare case which people usually discover on production, and go like “how can that be? It’s static!”

So, if your app is in background and system runs out of memory, it will start unloading classes (* see Update 3!). At some point, it may go ahead and unload Worker class. Next time you use it, it will be reloaded, and static variables will reset to default values. Since app itself never went out of memory entirely, there will be no call to Application.onCreate and no init(). Hence null context.

In other words, there’s absolutely no guarantee that Worker will initialize once per process. Every time you getInstance(), there’s a chance it was just recreated.

This issue is very poorly (read: not) documented in official documentation, so people are usually confused and scared about it. Then they discover that statics in App class are never unloaded like this (because App is guaranteed to live as long as process), and say “singleton yourselves, Android team, I don’t trust you now and will do the exact opposite of what you recommended”.

public class App extends Application {
    public static Worker worker;
    public void onCreate() {
        worker = new Worker();
        worker.init(this);
    }
}

This pattern will also work, and it kinda feels safer, because you don’t need to think about this reloading issue. But there are many cases when you’ll be perfectly fine with reloading, so you’ll just end up locking memory for no apparent reason.

Let’s check one more time, is this pattern affected by the reloading issue?

public class Worker {
    private static Worker instance = new Worker();
    private static Context context = App.getAppContext();
}

No, it’s not. If the class were reloaded, next time you call it, it will re-init its static members, so neither instance nor context will be null.

On the other hand, if Worker keeps some data you expect to change, you definitely have to make it an App static. Or, alternatively, persist the data every time.

public class Worker {
    Integer somedata;
    public void setSomedata(int value) {
        somedata = value;
        // save somedata to persistent storage like SharedPreferences
    }
    public int getSomedata() {
        if (somedata == null)
            // load somedata from persistent storage
        return somedata;
    }
}

I hope this post helps someone with all this confusing stuff.

PS: Updates

After this post was written, I’ve had interesting discussions on the subject and found that some of my statements were incorrect. Here are some updates on this.

Update 1: “final” matters

Notice that in previous examples I declare a lot of variables static, but not final. I never thought twice about it. Yes, they could potentially be changed, so I guess don’t change them. As it turns out, it does matter from the thread safety point of view. A “static final” variable is completely thread safe, while just “static” has to be synchronized. Read this StackOverflow and this Java document. So, correct example of my favorite singleton would look like this:

public class Worker {
    private static final Worker instance = new Worker();
    public static Worker getInstance() {
        return instance;
    }
}

Update 2: double-checked locking works

The issue with this pattern was related to operator reordering and specifics of “volatile” keyword. The article I read about this is quite old. So as it turns out, the “volatile” keyword was fixed in Java 5, and this pattern works now. It’s still gross and I would rather use the alternative, but there it is. See Wikipedia.

Update 3: classes do not unload (except sometimes they do)

Last time I researched this subject, I found a lot of anecdotal evidence that individual classes can be unloaded separately. I was relatively new to Java and it was generally weird to me, so I thought, that’s pretty disgusting, but okay. So now I took a second look, and it doesn’t really work like that. According to this post, it used to work exactly like that in some ancient Java, so that’s where the rumor comes from. In current version, specifically in the Android Java (whatever version that is), classes NEVER UNLOAD. We have that on good authority here.

But then again, the statement above is not the entire truth. Classes can in fact unload. But, only when all classes of a particular classloader decided to unload (aka were garbage collected). See this discussion. Also “JNI Tips“, which is an official document, mentions this case in an usual brief and confusing manner:

Classes are only unloaded if all classes associated with a ClassLoader can be garbage collected, which is rare but will not be impossible in Android.

Now, you might not know what a classloader is (which is normal and not a shame). Let’s just say java classes are also objects. Sounds like something a crazy person would say, but it makes sense to me. And there’s a collection or something where these objects are referenced. So when all objects are unused, and the collection is unused, logical step is to go ahead and garbage collect the entire thing.

Android apps happen to have two classloaders: a system one and an app’s one. I don’t know for sure why, I guess to make things complicated. But the app’s one basically references all your custom stuff. So if you don’t know about classloaders, don’t worry: only way yours will ever be unloaded is together with the entire app, after which point it will just be the same as a cold restart. You can however screw things up in an interesting way by using custom classloaders.

So, given all that, how come “null static” issue still exists? My guess is, it’s a combination of several scenarios described above. Like, doing stuff before App.onCreate(), or from an activity lifecycle event (and then the activity itself unloads and reloads in hilarious ways), or not realizing that classes with all their fields are in fact lazy-loaded. This post gives some more info (even though the author disses singletons, which I think is acrimonious of him). Also here and here are good practical examples.

Personally, I actually find it useful to think of a static as something that can be reloaded at any moment. Like, let’s say this class suddenly died and then re-inited, can it re-init its statics from some persistent storage or whatever? If the answer is no, then there’s a possibility for a problem.

How SMB works

One of the iOS projects I’ve been working on required communication with printers over SMB. As it turns out, iOS has absolutely no support for SMB. There are solutions for Linux, and Android, but porting any of these to iOS appeared to be quite an ordeal. After multiple attempts at that, I ended up implementing SMB myself. Then I extracted that code as a separate project (smb4ios). Here is some documentation from it, in case you’re interested in how SMB works.

First of all, what’s SMB. Initially (like in 1980’s) it was a protocol to access file shares. Since then, file shares evolved a lot, SMB however didn’t. Why is it still relevant then, you ask? Because it’s an integral part of Windows. There’s no “SMB standard”, there’s only “the way SMB works on Windows”. So, when we speak of SMB, we are interested not only in SMB itself (file system layer), but also in NetBIOS (transport layer), NTLM (authentication), and RPC. They all are kind of grown into each other by now, forming a monstrous layered ancient onion creature. So, from implementation perspective, you’ll have to deal with all of them simultaneously.

Let’s start with TCP. TCP is a low-level network protocol. When you want to send some data to network and other computers, you create a TCP socket, connect it to an address, and jam the data there. The socket takes care of packaging and delivery.

SMB is a transport protocol, kinda like HTTP. You should know that HTTP is basically when we send some extra information over TCP to describe what kind of content will be exchanged. SMB is the same, but over NetBIOS.

NetBIOS is basically a really old version of TCP. OSX and iOS do not have an implementation for it. Fortunately, it’s simple and can be implemented over TCP. So, this code is SMB over NetBIOS over TCP (aka NBT).

SMB was also called CIFS on some point, they are the same thing. There’s also SMB 2, which doesn’t work on Windows XP, so is irrelevant to me.

Now, first thing we need is to find the SMB servers in our network. That’s done by NetBIOS name queries. Basically it sends an UDP broadcast to a broadcast address (such as 255.255.255.255) and asks “who has this name?”. Then someone in the network may answer with his IP. There’s a main guy in NetBIOS network with a pre-defined name “master browser”, we ask for his IP first. Then we can ask him who else he knows, and that’s how we get domains and groups and server IPs.

Now that we have an IP, we can connect a TCP socket to it and run SMB messages.

First thing the server will want is for us to login under some user. You do the login by sending NTLM packets within SMB Session messages.

NTLM is when you use extremely complicated cryptography to encode and decode username, password etc. I used a 3rd-party library for that (see link below).

After login, you have basically opened a remote session with a Windows machine, and can run Windows API functions just like you normally would, if you were a Windows C++ application. You do that by sending RPC commands within SMB Transaction messages.

In short: RPC is a Windows API function code plus parameter buffer, which you wrap in SMB message. You can call functions like EnumAll, OpenPrinter, etc.

More details: Normally, to call a function with certain parameters, you need to push these parameters to stack. Which logically means all the neat C structures and variables you, the human, work with, are packed into a memory buffer and then that buffer is unpacked into the same pattern of structures and stuff inside the function (not what really happens). In RPC, you just provide the function code and the buffer. Which for some people means you should replicate the C structures and then pack them the same way. But in my opinion, nothing stops you from just dumping your data straight into bytes, as long as the end binary result is the same.

And that’s how it works: you find IP with NetBIOS, connect via SMB, send NTLM authentication, then run RPC commands. Use Wireshark, it’s a huge help, and I would be nowhere without it.

References:

SMB: [MS-SMB] at http://msdn.microsoft.com/en-us/library/cc246231.aspx
RAP: [MS-RAP] at http://msdn.microsoft.com/en-us/library/cc240190.aspx
DCE-RPC: http://pubs.opengroup.org/onlinepubs/9629399/chap12.htm
NetBIOS: RFC 1001 and 1002 (I dare you to actually read it and not go insane), http://www.ubiqx.org/cifs/NetBIOS.html
NTLM: http://www.nongnu.org/libntlm (the library I used)