How to select a Post Type (Wordpress) to pass a filter in head?

I have a WordPress site which uses Yoast SEO to manage canonical links and noindex meta robots, among other things, with a very peculiar configuration.

By default, Yoast's plugin adds a <link rel="canonical" target="_blank" rel="nofollow noreferrer" target="_blank" rel="nofollow noreferrer" href="..."> in every page. I'm not interested in show a canonical link in every single static page (whatever its type is) but in some type of posts: Pages and Posts (not Attatchment, Category, Archive, etc.)

I know the way to prevent Yoast from adding a canonical link by default, just adding the following code in functions.php: add_filter( 'wpseo_canonical', '__return_false' );. But what if I just want to pass that filter in some post types? Or passing it in every page except some types of posts (both approaches are useful to me). Help would be much appretiated. Any?

UPDATE: Valid answer with some minor fixes

function remove_canonical_from_post_types($url) {
    $post_type = get_post_type();
    if ($post_type !== 'post' && $post_type !== 'page') { // If post type is not Post nor Page, doesn't add a canonical link in any case
        return false;
    }
    else { // It is Post or Page
        if (is_category() || is_author() || is_tag() || is_archive()) { // If page is post type 'Post' we don't want to add canonical in some sub-types: Category, Author, Tag, Archive
            return false;
        }
        return $url; // In any other case (Posts and Pages) adds a canonical link
    }
}
add_filter('wpseo_canonical', 'remove_canonical_from_post_types');

Try to use the provided "wpseo_canonical" filter and check there for the correct post type (and return false there).

Something like this:

function remove_canonical_from_post_types($url)
{
    $post_type = get_post_type();

    if ($post_type !== 'post' && $post_type !== 'page') {
        return false;
    }

    return $url;
}
add_filter('wpseo_canonical', 'remove_canonical_from_post_types');