October 20, 2013

Dalvik

This is another really big topic ūüôā Many people out there talk about Dalvik but according to many conversations I hold, it is not very clear to them, what exactly it is. The next is the best and more simple definition I found about Dalvik:

Dalvik is Android’s Java virtual machine. It allows Android to run the byte-code generated from Java-based apps and Android’s own system components and provides both with the required hooks and environment to interface with the rest of the system, including native libraries and the rest of the native user-space.¬†– Embedded Android

But there are some differences between Dalvik, from now on DVM, and the Java Virtual Machine, JVM.

Dalvik was design for embedded systems. Those systems would have a small amount of RAM, slow CPU and run an OS without swap space. Also they would be battery powered. As you probably know, if you want to run a Java application, the JVM is loaded with the necessary .class flies via a .jar file. In case of Dalvik there are no such .class files, instead there is a single .dex file which is 50% smaller than the original .jar file. DEX stands for Dalvik Executable. DVM is register-based machine, different from the push-pop stack-based JVM. Another important difference is that DVM includes a Just In Time (JIT) compiler for ARM. This basically means that the code is compiled at run time so that it can run natively in the CPU instead of been interpreted since that is slower. This was already typical in other virtual machines but it makes much more sense in the kind of systems for what Dalvik was design. Traditionally most of the Java code was run in the server side by very powerful machines but as technology evolves we have more code running in the client side and this kind of optimizations are more of a necessity. Another important difference between DVM and JVM is that the JVM relies on the Java Development Kit (JDK), while DVM relies on the Apache Harmony project, which is a Java implementation hosted by the Apache project. The JDK is used in AOSP only for the java compiler at build time but none of those are distributed with Android.

DEX files might look like harder to decompile than the traditional JAR files but it is not. We will check that in later posts on how to decompile Android applications. That topic is very related to this post since we will have the opportunity to go mote into Dalvik internals.

If you go to the Zygote post, you will see that Zygote was in charge of spawning the resources that applications need to run. One of the tasks is to fork an instance of a DVM so that the new application can run on it. Bellow you can see how Zygote forks a new instance of the DVM. This new process is managed by the Activity Manager as we previously saw.

/**
 * Forks a new VM instance. The current VM must have been started
 * with the -Xzygote flag. <b>NOTE: new instance keeps all
 * root capabilities. The new process is expected to call capset()</b>.
 *
 * @param uid the UNIX uid that the new process should setuid() to after
 * fork()ing and and before spawning any threads.
 * @param gid the UNIX gid that the new process should setgid() to after
 * fork()ing and and before spawning any threads.
 * @param gids null-ok; a list of UNIX gids that the new process should
 * setgroups() to after fork and before spawning any threads.
 * @param debugFlags bit flags that enable debugging features.
 * @param rlimits null-ok an array of rlimit tuples, with the second
 * dimension having a length of 3 and representing
 * (resource, rlim_cur, rlim_max). These are set via the posix
 * setrlimit(2) call.
 * @param seInfo null-ok a string specifying SEAndroid information for
 * the new process.
 * @param niceName null-ok a string specifying the process name.
 *
 * @return 0 if this is the child, pid of the child
 * if this is the parent, or -1 on error.
 */
 public static int forkAndSpecialize(int uid, int gid, int[] gids,
         int debugFlags, int[][] rlimits, int mountExternal, String seInfo, String niceName) {
     preFork();
     int pid = nativeForkAndSpecialize(
             uid, gid, gids, debugFlags, rlimits, mountExternal, seInfo, niceName);
     postFork();
     return pid;
 }

If you want and I recommend you to do it, you can check the nativeForkAndSpecialize routine to the native code you can find it in the Google Code here

With all this, there are associated tools supported by the platform and which you can use to play around from an system shell ūüôā

  • dalvikvm: Starts a clean DVM with no Android functionality. You can load run .jar files.
  • dvz: Starts a new DVM, but this time with Android functionality. The difference with respect to the way Zygote to it is that this would be a detached process not managed by Activity Manager
  • dexdump: It helps decompiling DEX files. This will be related to future posts on how to decompile Android applications.

And now, if you want to go and check more, here is Dan Bornstein giving a brief on the topic: http://www.youtube.com/watch?v=ptjedOZEXPM

Advertisements
October 16, 2013

Activity Manager

In the last post we talked about Zygote, and how that special process launched applications. But for Zygote to launch an application is necessary to be requested. Users request to launch an application via a tap on an App icon from the home screen. The home screen as users know it is an application and the only one listening for onClick(). When this happens the launcher contact the ActivityManager. That is, request a handler through the Binder and call startActivity() method from the Activity Manager. If you remember from the Binder post, when an application wants to contact a remote method from another service, it first request a handler to the Binder, this will ask for that service to the Service Manager and pass that handler to the originator. When the Activity Manager receives the request to startActivity() it contacts Zygote to fork itself and start that new VM to run the application. It does this via startViaZygote() method which will open a connection to the Zygote socket (/dev/socket/zygote).

Now you can see how all the previous posts fall together in Android, cool eh ūüôā (images: from Embedded Android)

Android Boot sequence

Android Boot sequence (from Embedded Android)

Service Manager and Binder Interaction

Service Manager and Binder Interaction (from Embedded Android)

But the Activity Manager not only do this. It is responsible for many other tasks. This service is responsible for such an important task as broadcasting intents. You probably saw that scary pop up message saying “Application Not Responding” aka ANR. Then you should know that the Activity Manager is the responsible for this. Not only that, it also sends the intent INTENT.CATEGORY_HOME once it finish booting and reports System Ready, broadcasting the rest of the system this information. If you check the documentation of the service (http://developer.android.com/reference/android/app/ActivityManager.html), you will see the available public methods and make yourself an idea of how much this guy is doing.

This service is tremendously big, is huge and has lots of tasks to talk about. This is an the introductory post to the service but we will certainly come back to those tasks and see on code how are done. In the meanwhile you might want to talk a look to the public code, you can located in google source. That exposes the public API.

Embedded Android

October 15, 2013

Zygote

Now, that’s a name. Zygote is a daemon whose goal is to launch Apps. The startup of the process is triggered by Init.rc¬†after Service Manager and others but it is actually started by app_process. Here is the sequence used to start such special process

service zygote /system/bin/app_process -Xzygote /system/bin --zygote --start-system-server
class main
socket zygote stream 660 root system
onrestart write /sys/android_power/request_state wake
onrestart write /sys/power/state on
onrestart restart media
onrestart restart netd

You can see how Zygote is started as a system service. You can see also how app_process is the actual command that starts it.

As we said, Zygote is a daemon which only mission is to launch applications. This means that Zygote is the parent of all App process. When app_process launches Zygote, it creates the first Dalvik VM and calls Zygote’s main () method. Once Zygote starts, it preloads all necessary Java classes and resources, starts System Server and opens a socket /dev/socket/zygote to listen for requests for starting applications. System Server as we saw in previous post is a complete detached process from it’s parent. Once it’s created it goes initializing all the different System Services¬†and starts the Activity Manager (will see this in a later posts).¬†So, how does Zygote start new applications?

Zygote receives a request to launch an App through /dev/socket/zygote. Once it happens it trigger a fork() call. Here is where the virtues of a great design take action. When a process forks, it creates a clone of itself. It replicates itself in another memory space. This is done pretty efficiently. When this happens to Zygote, it creates an exact and clean new Dalvik VM, preloaded with all necessary classes and resources that any App will need. This makes the process of creating a VM and load resources pretty efficiently. But the design goes farther. As we know, Android runs on Linux. The Linux Kernel implements a strategy call Copy On Write (COW). What this means is that during the fork process, no memory is actually copy to another space. It is shared and marked as copy-on-write. Which means that when a process attempt to modify that memory, the kernel will intercept the call and do the copy of that piece of memory. In the case of Android those libraries are not writable. This means that all process forked from Zygote are using the exact same copy of the system classes and resources. Another benefit is the real memory saving. No matter how many applications are started the increase in memory usage will be a lot smaller.

If you are in an early stage of a porting process, Zygote is another big process that you don’t need to start. You can disable it by adding the keyword “disabled” at the end of the boot sequence in¬†init.rc

service zygote /system/bin/app_process -Xzygote /system/bin --zygote --start-system-server
class main
socket zygote stream 660 root system
onrestart write /sys/android_power/request_state wake
onrestart write /sys/power/state on
onrestart restart media
onrestart restart netd
disabled

AlBelow is a picture (from Embedded Android) on where can you place Zygote in the Android Platform

zygote

Android Boot Sequence (from Embedded Android)

Here is the Zygote main() method:

public static void main(String argv[]) {
 try {
     // Start profiling the zygote initialization.
     SamplingProfilerIntegration.start();

    registerZygoteSocket();
    EventLog.writeEvent(LOG_BOOT_PROGRESS_PRELOAD_START,
    SystemClock.uptimeMillis());
    preload();
    EventLog.writeEvent(LOG_BOOT_PROGRESS_PRELOAD_END,
    SystemClock.uptimeMillis());

    // Finish profiling the zygote initialization.
    SamplingProfilerIntegration.writeZygoteSnapshot();

    // Do an initial gc to clean up after startup
    gc();

    // Disable tracing so that forked processes do not inherit stale tracing tags from
    // Zygote.
    Trace.setTracingEnabled(false);

    // If requested, start system server directly from Zygote
    if (argv.length != 2) {
        throw new RuntimeException(argv[0] + USAGE_STRING);
    }

    if (argv[1].equals("start-system-server")) {
        startSystemServer();
    } else if (!argv[1].equals("")) {
        throw new RuntimeException(argv[0] + USAGE_STRING);
    }

    Log.i(TAG, "Accepting command socket connections");

    runSelectLoop();

    closeServerSocket();
 } catch (MethodAndArgsCaller caller) {
     caller.run();
 } catch (RuntimeException ex) {
     Log.e(TAG, "Zygote died with exception", ex);
     closeServerSocket();
     throw ex;
 }
}

Also, see below how Zygote starts the System Server:

/**
 * Prepare the arguments and fork for the system server process.
 */
 private static boolean startSystemServer()
 throws MethodAndArgsCaller, RuntimeException {
 /* Hardcoded command line to start the system server */
 String args[] = {
 "--setuid=1000",
 "--setgid=1000",
 "--setgroups=1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1018,3001,3002,3003,3006,3007",
 "--capabilities=130104352,130104352",
 "--runtime-init",
 "--nice-name=system_server",
 "com.android.server.SystemServer",
 };
 ZygoteConnection.Arguments parsedArgs = null;

int pid;

try {
 parsedArgs = new ZygoteConnection.Arguments(args);
 ZygoteConnection.applyDebuggerSystemProperty(parsedArgs);
 ZygoteConnection.applyInvokeWithSystemProperty(parsedArgs);

/* Request to fork the system server process */
 pid = Zygote.forkSystemServer(
 parsedArgs.uid, parsedArgs.gid,
 parsedArgs.gids,
 parsedArgs.debugFlags,
 null,
 parsedArgs.permittedCapabilities,
 parsedArgs.effectiveCapabilities);
 } catch (IllegalArgumentException ex) {
 throw new RuntimeException(ex);
 }

/* For child process */
 if (pid == 0) {
 handleSystemServerProcess(parsedArgs);
 }

return true;
 }
October 12, 2013

Android Framework

The Android Framework is not a specific component of the Android platform. It is a very broad naming of a specific layer in all the Android architecture. In the picture below (from Embedded Android), I like to place the Android Framework in the android.* rectangle. But other people also include System Services, Android Runtime and some native daemons. Binder and User Apps and Stock Android Apps. Sometimes even Dalvik and Zygote. Basically everything that you can find in the frameworks directory in the AOSP (plus something more :).

As you can see the Android Framework is what most people know as Android. It is where most developers base their applications on. The Android Platform includes also other frameworks to help and support developers, like Apache Harmony, but that is not part of the Android Framework. Those are externals.

AndroidArquitechture

AndroidArquitechture (from Embedded Android)

Since Android Framework is just those Dev APIs, you can imagine that you can build AOSP without the framework. It might look weird but Android can be suitable for other kind of systems that is not running a hand-held with Apps. AOSP provides a specific configuration in the build system for this. To do this, just do the next

$ BUILD_TINY_ANDROID=true make -j16

This is what is called Tiny Android. It is specially useful if you are in an early stage of a porting process to a new platform. It will build Init, Bionic, Toll-box, adbd, logcat, and some other binaries and libraries.

So, as a resume, there is a bit of confusion out there on what really is the Android Framework. You can check the sources here and see it by yourself. As you see in the picture the frameworks rely on many services and key components, some of them we already discussed in previous posts like Service Manager, and others we did not discussed yet. But we will in later posts, like Zygote, Surfaceflinger, Android Runtime and Dalvik.

AndroidArquitechture

October 11, 2013

Service Manager

The Service Manager is like an information director. It is the Yellow Pages of all services available. It has a very close relationship with the Binder. In the Binder post you can see a picture with the Service Manager, and you can see that when an application wants to instantiate a service they first have to get a handler. That is, they have to ask the Service Manager for that handler, and they do it through the Binder.

It make sense then that the Service Manager is started before any other service. It is actually started by Init. At that time, the Service Manager request the Binder to be the “Binder Context Manager”, which is like to say I want to be the secretary of the boss. It does this through an ioctl() call. As we saw in the System Services the System Server will add all the spawn services to the Service Manager.

As you can imagine the Service Manager is not really visible by the developer, and most are unaware of it’s existence, but it is there to find your service. If Service Manager is the index to all the other System Service it makes sense that is tarted before any other. For that to happen there is only one place for this to happen, and that is in *init.rc*.¬† After it, follow Zygote, meda, surfaceflinger and drm. These are the first and most basic services started and they are natively implemented. The reason as we said in the first post, they have to be highly CPU efficient.

service servicemanager /system/bin/servicemanager class core user system group system critical
onrestart restart zygote
onrestart restart media
onrestart restart surfaceflinger
onrestart restart drm

You can see here ServiceManager.java, interface which implements the Java IServiceManager.java Interface. These are the method implemented at the Service Manager. You can find here IServiceManager.cpp the native code for this service, that is, the real service. It might look confusing. You should get familiar with Java Interfaces if you are not. It is very normal in Android. Each service has normally an Interface which is implemented in a class. The class can be the Java interface of a native service or a Java implemented service. Following you can see the Interface (you can not instantiate) of the ServiceManager to illustrate the methods implemented by the service.

public interface IServiceManager extends IInterface
{
/**
* Retrieve an existing service called @a name from the
* service manager.  Blocks for a few seconds waiting for it to be
* published if it does not already exist.
*/
public IBinder getService(String name) throws RemoteException;

/**
* Retrieve an existing service called @a name from the
* service manager.  Non-blocking.
*/
public IBinder checkService(String name) throws RemoteException;
/**
* Place a new @a service called @a name into the service
* manager.
*/
public void addService(String name, IBinder service, boolean allowIsolated)
throws RemoteException;
/**
* Return a list of all currently running services.
*/
public String[] listServices() throws RemoteException;
/**
* Assign a permission controller to the service manager.  After set, this
* interface is checked before any services are added.
*/
public void setPermissionController(IPermissionController controller)
throws RemoteException;

static final String descriptor = "android.os.IServiceManager";
int GET_SERVICE_TRANSACTION = IBinder.FIRST_CALL_TRANSACTION;
int CHECK_SERVICE_TRANSACTION = IBinder.FIRST_CALL_TRANSACTION+1;
int ADD_SERVICE_TRANSACTION = IBinder.FIRST_CALL_TRANSACTION+2;
int LIST_SERVICES_TRANSACTION = IBinder.FIRST_CALL_TRANSACTION+3;
int CHECK_SERVICES_TRANSACTION = IBinder.FIRST_CALL_TRANSACTION+4;
int SET_PERMISSION_CONTROLLER_TRANSACTION = IBinder.FIRST_CALL_TRANSACTION+5;
}
October 3, 2013

System Services

The System Services is a set of about 60-80 services that provide the user applications with the information and capabilities necessary to work. All these services relay on the Binder to communicate and collaborate with each other. Also, applications can instantiate these services through Binder. Most are written in Java except for a couple of them, which are written in C/C++. The reason is that those C/C++ services relay on hardware acceleration or are in need or a high performance implementation. A good example is the sensor service or the surface flinger. Other services like camera service or media player, make an extensive use of Native Code through JNI calls.

System Server

Among all the services we have to highlight one of them, the System Server. What does make so special to this guy? This guy is the ruler, I like to call it, the Darth Vader of the System. This service starts all the other services, but not only that, it also bootstraps the file infrastructure, with critical directories like /data/user. It creates each service and add it to the Service Manager. Most of those services are Java-built as you can see in the picture below. This services are JNI friendly ūüôā , so don’t be surprise to find JNI calls to Native code.

System Services

System Services (from Embedded Android)

If you want your android to boot faster, you only have to comment out some of those services. Also, you might want to substitute some of those services for your own implementation. If you want your own Wallpaper Service for example, you can modify the service itself or substitute the whole thing. Modify it would be easer, though. You can also make some Java service Native.

Following is a little snippet of actions that take place in System Server (SystemServer.java)

class ServerThread extends Thread {
    private static final String TAG = "SystemServer";
    private static final String ENCRYPTING_STATE = "trigger_restart_min_framework";
    private static final String ENCRYPTED_STATE = "1";

    ContentResolver mContentResolver;

    void reportWtf(String msg, Throwable e) {
        Slog.w(TAG, "***********************************************");
        Log.wtf(TAG, "BOOT FAILURE " + msg, e);
    }

    @Override
    public void run() {
        EventLog.writeEvent(EventLogTags.BOOT_PROGRESS_SYSTEM_RUN,
        SystemClock.uptimeMillis());

        Looper.prepareMainLooper();

        android.os.Process.setThreadPriority(
        android.os.Process.THREAD_PRIORITY_FOREGROUND);

...

        Slog.i(TAG, "Battery Service");
        battery = new BatteryService(context, lights);
        ServiceManager.addService("battery", battery);

        Slog.i(TAG, "Vibrator Service");
        vibrator = new VibratorService(context);
        ServiceManager.addService("vibrator", vibrator);

...

        Slog.i(TAG, "Window Manager");
        wm = WindowManagerService.main(context, power, display, inputManager, uiHandler, wmHandler,
                               factoryTest != SystemServer.FACTORY_TEST_LOW_LEVEL,
                               !firstBoot, onlyCore);

        ServiceManager.addService(Context.WINDOW_SERVICE, wm);
        ServiceManager.addService(Context.INPUT_SERVICE, inputManager);

        ActivityManagerService.self().setWindowManager(wm);

        inputManager.setWindowManagerCallbacks(wm.getInputMonitor());
        inputManager.start();

        display.setWindowManager(wm);
        display.setInputManager(inputManager);

...

I will not talk about Media Service by now, that requires an extensive posts and I don’t want to make this too heavy.
Next day we will talk about the Services Manager.

October 2, 2013

The Binder (I)

High level view on the Android Binder

The Binder¬†is one of the most interesting mechanisms in Android. Binder is an RPC/IPC mechanism. It might be surprising to many but the concept is not an Android innovation. It’s roots date back to BeOS before it was acquire by Palm. That Binder code was released as Open Source weeks before Palm’s death and named as OpenBinder. The project was started by George Hoffman and continued by¬†Dianne Hackborn¬†and the team at PalmSource. She was the one who actually opened it to The World before it disappeared. In any case, Android’s Binder is not a fork of OpenBinder but it is an implementation of that concept from the ground up. Yeap Dianne works at Android now ūüôā

If you want to know more about the original OpenBinder, you should go and read the Documentation (http://www.angryredplanet.com/~hackbod/openbinder), it remains a must-read. 

The Binder in Android is like the Rosette Stone. It allows Apps to talk to System Server and it enable Apps to talk to each other. Developers don’t need to know any of this. They just use different interfaces for the services but under the hood, those interfaces use the Binder under the hood. Developers just don’t see it.

Android Binder

Android Binder (from Embedded Android)

As you can see in the picture above, The Binder (/dev/binder), is placed in the middle of all the action and it is used in almost everything that happens across processes in the core platform.

For example, let say an application wants to instantiate the Window Service, the next line would be the necessary code:

WindowManager mWindowManager = (WindowManager)context.getSystemService(Context.WINDOW_SERVICE)

The application now can use all WindowManager services like:

  • mWindowManager.addView(mDecor, l);
  • mWindowManager.removeView(mDecor);
  • mWindowManager.updateViewLayout(mDecor, params);
  • ...

When an application instantiates a Service, that process is going through the Binder to the Service Manager, look it up, and sends a reference to the App, then the App can go and requests remote calls from that object. like updateViewLaout(). If you need to know more about IPC/RPC I recommend you to read a¬†developer’s introduction to RPC and XDR, from SGI IRIX documentation.

This is overall the first part of this Binder posts. The Binder is too complex to talk about it in just one post so lets keep them dropping step by step.

Tags: , ,
October 1, 2013

Android root Certificates Management

This post is not about root certificates. It is not analyzing the different formats of security certificates neither how to generate those.

This post is about how Android deals with security certificates and how to change it to make it more secure. Android v4.1+

There are mainly two source files you want to check to understand this better (NativeCrypto.java and TrustedCertificateStore.java). This is all part of the Apache Harmony for Android Open Source Project.

Android, as any other mobile operating system has a CA root security certificates store that uses to validate the identity of remote peers. It is the same way as an internet browsers does. There is a directory (/etc/security/cacerts), where Android stores the root certificates. This is called the Certificates Store. When an application contacts a remote server, this sends it’s identity to the Android device. Then Android validates it or rejects it based on the trusted certificates stored in the Certificates Store.

Root Certificates are formatted in PEM. They include also the human readable version of it. The way Android name these certificates is by using an MD5 hash, based on the unique subject name field of the certificate. This means that the name of the file for the certificate with Subject field equals to “C=US, O=GeoTrust Inc., CN=GeoTrust Global CA” is 7999be0d.0

As we can see in the next code from TrustedCertificateStore.java, the search method will first request the MD5 hash and then will perform the search on the Certificates Store.


private String hash(X500Principal name) {
    int hash = NativeCrypto.X509_NAME_hash_old(name);
    return IntegralToString.intToHexString(hash, false, 8);
}

How to make this more safe

There are some things you can do if you want your Android to be more safe. And there are reasons for this.

MD5 hash algorithm is broken. It is surprising that Google still uses this MD5 hash algorithm. This is something that is totally broken and nobody should use it anymore. Apache Harmony provides another method to calculate the SHA1 instead of MD5.  To do this:

  1. Use NativeCrypto.X509_NAME_hash(name); instead of NativeCrypto.X509_NAME_hash_old(name);
  2. Rename all your certificates to use a SHA1 based name. Watch out here if you use openssl to do this. Openssl generates a different SHA1 hash than the Apache Harmony framework. Why? Apache Harmony bases the MD5 and SHA1 hash in the Subject name of the certificate. Before computing the hash, it transform that subject string into a ASN1 DER format. This is done for both MD5 and SHA1 and is pretty standard across different frameworks.

The openssl command line tool does not behaves like that. Encryption is like money laundry, the more hands you can pass the money through, the harder in theory to track it back to the originator. Openssl folks correctly believed that SHA1 was not safe enough so they decided to pass it through one more filter. It first calculates the CANONICAL format of the string. Then you take that String and generate the ASN1 DER format of it. Then it passes the encoded string to the SHA1 algorithm to calculate the digest. Yeap, pretty nasty. I believed that openssl.1.0.1e libraries already include all these changes. But if you are using older ones you will have to modify them to be able to programmatically calculate the SHA1 of a certificate. You can read a little bit more about how to do this in this thread from the openssl mailing list

I hope that eventually, better sooner than later, Google decides to go for at least SHA1. At this point SHA1 is no safe either but hey, it’s better than MD5.

September 30, 2013

Hello Android!

Hello Android!

This Blog is a way for me to share my knowledge on Android. I will comment on how Android manage things internally. How the internal servers and services communicate and get integrated together. Also, how to port Android to different platforms. How to decompile applications to find security threats or improve performance. Maybe somebody else out there can use these posts to resolve same problems I had. Please do not hesitate to comment or contact me for doubts or suggestions. anatomyofandroid [at] gmail [.] com

GoogleAnatomyMedium

Let’s start dissecting this robot!