Laravel Sorting 1st Level Collection By 2nd Level Value

I have users with (or without) posts and I want the order of users to be dictated by the last post (showing the users who have never posted, then posted the longest ago first).

User A has posted yesterday
User B posted a week ago.
User C has never posted.

The order of users should be C,B,A.

The following code is how far I've come, but I can't seem to get it right. The order of users is not as I would expect it to be. Any thoughts?

UPDATE:
After removing the limit() on eager-load, The result of sorting is: C,A,B, instead of C,B,A.

$users = User::with('roles', 'flags')->whereHas('subscription', function($x) use($category_id) {
    $x->where('subscription.category_id', '=', $category_id);
})->get()->load([
    'posts' => function($y) {
        return $y->orderBy('created_at', 'ASC');
    }
]);

$users = $users->sortBy(function($user) {
    if(count($user->posts) == 0) {
        return 0;
    }

    return strtotime($user->posts[0]->created_at);
});

return $users;

Thanks to @patricus in the comment section, my final (working) code is:

$users = User::with('roles', 'flags')->whereHas('subscription', function($x) use($category_id) {
    $x->where('subscription.category_id', '=', $category_id);
})->get()->load([
    'posts' => function($y) {
        return $y->latest();
    }
]);

$users = $users->sortBy(function($user) {
    if(count($user->posts) == 0) {
        return 0;
    }

    return strtotime($user->posts[0]->created_at);
});

return $users;