By default the Insight Agent attempts to validate the SSL certificate being presented by the server, which in your case will be the default self-signed certificate delivered out of the box.
The Insight Agent configuration policy template (ADMX) referenced in the quick start guide, has a setting called "SSL Cert. Flags" which allows you to configure the client to ignore SSL validation errors. Setting this should allow your client to successfully talk to the appliance.
If you've set this item and are still seeing this error, this might indicate a mismatch between the SSL TLS version supported by the Insight server and your client operating system, in which case please have a look at the following article to see if TLS v1.1 and 1.2 are enabled on your client.
Note: The article references a different product (Environment Manager) but the Microsoft KB article and example registry items applies in this case too:
Let me know how you get on.
I told you i am very newbie to insight
I could not find The Insight Agent configuration policy template (ADMX) in support.
I just change my reqistry setting
BTW, Insight and my machine is not in domain.
The template can be downloaded from here: ADMX Templates for Insight v10
Alternatively, you can add the Certificate setting to ignore errors directly to the registry (this is ultimately what the ADMX writes when you configure the setting):
Value Type: DWORD
Value Data: 3300
The machines not being domain joined is no problem. If you'd like to use the ADMX on your client you can do so via the local policy editor.