How do I set up criteria for bootloader to decide if it shall run the bootloader or the application at the startup?

For this you need to have a decision criterion, which is typically a jumper or a push button to be pressed at power up to enter bootloader mode. This is performed by the function BL_CheckForUserApp():

/*!
 * \brief This method is called during startup! It decides if we enter bootloader mode or if we run the application.
 */
void BL_CheckForUserApp(void) {
  uint32_t startup; /* assuming 32bit function pointers */

  startup = ((uint32_t*)APP_FLASH_VECTOR_START)[1]; /* this is the reset vector (__startup function) */
  if (startup!=-1 && !BL_CheckBootloaderMode()) { /* we do have a valid application vector? -1/0xfffffff would mean flash erased */
    ((void(*)(void))startup)(); /* Jump to application startup code */
  }
}

It first fills a callback structure of type S19_ParserStruct, as the following listing shows:

typedef struct S19_ParserStruct {
  uint8_t (*GetCharIterator)(uint8_t*, void*); /* character stream iterator */
  void *voidP; /* void pointer passed to iterator function */
  uint8_t (*S19Flash)(struct S19_ParserStruct*); /* called for each S19 line to be flashed */
  /* the following fields will be used by the iterator */
  S19_FileStatus status; /* current status of the parser */
  uint8_t currType; /* current S19 record, e.g. 1 for S1 */
  uint32_t currAddress; /* current code address of S19 record */
  uint16_t codeSize; /* size of code in bytes in code buffer */
  uint8_t *codeBuf; /* code bufffer */
  uint16_t codeBufSize; /* total size of code buffer, in bytes */
} S19_ParserStruct;

This function checks if the startup function in the vector table (index 1) is valid or not. If the application flash has been erased, it reads -1 (or 0xffffffff). So if you have an application present and the user does not want to run the bootloader, we jump to the application startup.

Listed below is the code to decide that on pressing the button, bootloader enters the startup code:

static bool BL_CheckBootloaderMode(void) {
  /* let's check if the user presses the BTLD switch. Need to configure the pin first */
  /* PTB8 as input */
  /* clock all port pins */
  SIM_SCGC5   |= SIM_SCGC5_PORTA_MASK |
             SIM_SCGC5_PORTB_MASK |
             SIM_SCGC5_PORTC_MASK |
             SIM_SCGC5_PORTD_MASK |
             SIM_SCGC5_PORTE_MASK;
  /* Configure pin as input */
  (void)BitIoLdd3_Init(NULL); /* initialize the port pin */
  if (!BL_SW_GetVal()) { /* button pressed (has pull-up!) */
    WAIT1_Waitms(50); /* wait to debounce */
    if (!BL_SW_GetVal()) { /* still pressed */
      return TRUE; /* go into bootloader mode */
    }
  }
  /* BTLD switch not pressed, and we have a valid user application vector */
  return FALSE; /* do not enter bootloader mode */
}