Laravel Polymorphic Many-to-many Not Detaching?

I have a polymorphic many to many User/Location - Group relationship, but I'm having a time getting the basic unit testing to pass.

/** @test */
public function it_can_have_groups()
{
    $user = factory(User::class)->create();
    $group = factory(Group::class)->create();

    $user->addToGroup($group);
    $this->assertCount(1, $user->groups);  // true

    $user->removeFromGroup($group);
    $this->assertCount(0, $user->groups);  // false
}

These methods simply call attach and detach as per the documentation:

To remove a many-to-many relationship record, use the detach method. The detach method will delete the appropriate record out of the intermediate table; however, both models will remain in the database

public function addToGroup(Group $group)
{
    $this->groups()->attach($group->id);
}

public function removeFromGroup(Group $group)
{
    $this->groups()->detach($group->id));
    dump($group->users());                 // []
}

So it seems to work (?), but only from the groups side, and the assertion still fails. Why, and what should I be doing differently?

It may be as simple as the model you are testing having stale relationships. Try this:

/** @test */
public function it_can_have_groups()
{
    $user = factory(User::class)->create();
    $group = factory(Group::class)->create();

    $user->addToGroup($group);
    $this->assertCount(1, $user->groups->fresh());  // true

    $user->removeFromGroup($group);
    $this->assertCount(0, $user->groups->fresh());  // false
}